Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250624 07:54:53.197016 13008 test_util.cc:276] Using random seed: -1530704208
2025-06-24T07:54:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T07:54:53Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskPG5ypB/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskPG5ypB/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/security.properties
2025-06-24 07:54:58: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskPG5ypB/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskPG5ypB/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250624 07:55:06.730141 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--webserver_interface=127.12.180.62
--webserver_port=0
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250624 07:55:07.047039 13249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:07.047372 13249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:07.047444 13249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:07.054811 13249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:07.054965 13249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:07.055022 13249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:07.055096 13249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:07.064404 13249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.180.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:07.066151 13249 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:07.068094 13249 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:07.078538 13255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:07.078636 13257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:07.078586 13254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:07.079226 13249 server_base.cc:1048] running on GCE node
I20250624 07:55:07.080042 13249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:07.081403 13249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:07.082582 13249 hybrid_clock.cc:648] HybridClock initialized: now 1750751707082495 us; error 79 us; skew 500 ppm
I20250624 07:55:07.082921 13249 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:07.089649 13249 webserver.cc:469] Webserver started at http://127.12.180.62:45479/ using document root <none> and password file <none>
I20250624 07:55:07.090481 13249 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:07.090574 13249 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:07.090855 13249 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:55:07.093113 13249 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/instance:
uuid: "588e956d4ac040deb3c06e57bd3c6b88"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.093775 13249 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal/instance:
uuid: "588e956d4ac040deb3c06e57bd3c6b88"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.098925 13249 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20250624 07:55:07.102079 13263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:07.103412 13249 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250624 07:55:07.103545 13249 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
uuid: "588e956d4ac040deb3c06e57bd3c6b88"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.103735 13249 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:07.116801 13249 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:07.117615 13249 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:07.117861 13249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:07.136631 13249 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.62:44637
I20250624 07:55:07.136648 13314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.62:44637 every 8 connection(s)
I20250624 07:55:07.138247 13249 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
I20250624 07:55:07.141753 13315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:07.147671 13315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } has no permanent_uuid. Determining permanent_uuid...
I20250624 07:55:07.148329 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 13249
I20250624 07:55:07.148625 13008 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal/instance
I20250624 07:55:07.151564 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.61:46565
--webserver_interface=127.12.180.61
--webserver_port=0
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250624 07:55:07.164249 13315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } has no permanent_uuid. Determining permanent_uuid...
W20250624 07:55:07.165498 13266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:07.167053 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:07.216398 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } attempt: 1
W20250624 07:55:07.218334 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:07.286597 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } attempt: 2
W20250624 07:55:07.288576 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:07.379879 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } attempt: 3
W20250624 07:55:07.381763 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:07.459062 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:07.459367 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:07.459441 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:07.466881 13318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:07.466984 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:07.467038 13318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:07.467083 13318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:07.476297 13318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.61:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.180.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:07.477815 13318 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:07.479519 13318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:07.489380 13327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:07.489413 13324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:07.489343 13325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:07.490156 13318 server_base.cc:1048] running on GCE node
I20250624 07:55:07.490892 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:07.492228 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:07.493441 13318 hybrid_clock.cc:648] HybridClock initialized: now 1750751707493385 us; error 28 us; skew 500 ppm
I20250624 07:55:07.493757 13318 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:07.495952 13318 webserver.cc:469] Webserver started at http://127.12.180.61:43475/ using document root <none> and password file <none>
I20250624 07:55:07.496692 13318 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:07.496798 13318 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:07.497133 13318 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:55:07.499661 13318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/instance:
uuid: "52afd679ffb241a3a9022cc850c03493"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.500425 13318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal/instance:
uuid: "52afd679ffb241a3a9022cc850c03493"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.505254 13318 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20250624 07:55:07.508383 13333 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:07.509593 13318 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250624 07:55:07.509737 13318 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
uuid: "52afd679ffb241a3a9022cc850c03493"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.509914 13318 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:07.525018 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } attempt: 4
W20250624 07:55:07.526685 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:07.533274 13318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:07.534080 13318 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:07.534338 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:07.553057 13318 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.61:46565
I20250624 07:55:07.553076 13384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.61:46565 every 8 connection(s)
I20250624 07:55:07.554445 13318 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
I20250624 07:55:07.558137 13385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:07.559754 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 13318
I20250624 07:55:07.559932 13008 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal/instance
I20250624 07:55:07.560870 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--webserver_interface=127.12.180.60
--webserver_port=0
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250624 07:55:07.563833 13385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } has no permanent_uuid. Determining permanent_uuid...
I20250624 07:55:07.576351 13385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } has no permanent_uuid. Determining permanent_uuid...
I20250624 07:55:07.584251 13385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } has no permanent_uuid. Determining permanent_uuid...
W20250624 07:55:07.585265 13334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:07.586656 13385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:07.635978 13385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 1
W20250624 07:55:07.637885 13385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:07.706102 13385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 2
W20250624 07:55:07.708021 13385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:07.799281 13385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 3
W20250624 07:55:07.801116 13385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:07.825955 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } attempt: 5
I20250624 07:55:07.831887 13315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } has no permanent_uuid. Determining permanent_uuid...
W20250624 07:55:07.833518 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:07.863529 13386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:07.863845 13386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:07.863967 13386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:07.871870 13386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:07.871984 13386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:07.872049 13386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:07.872124 13386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:07.881632 13386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.180.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:07.883224 13386 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:07.884735 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 1
I20250624 07:55:07.884915 13386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:07.886524 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:07.895068 13397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:07.895222 13394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:07.895125 13395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:07.896102 13386 server_base.cc:1048] running on GCE node
I20250624 07:55:07.896778 13386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:07.898082 13386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:07.899286 13386 hybrid_clock.cc:648] HybridClock initialized: now 1750751707899248 us; error 28 us; skew 500 ppm
I20250624 07:55:07.899576 13386 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:07.901648 13386 webserver.cc:469] Webserver started at http://127.12.180.60:42595/ using document root <none> and password file <none>
I20250624 07:55:07.902408 13386 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:07.902494 13386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:07.902777 13386 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:55:07.905012 13386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/instance:
uuid: "632ba412a681447e93af32d7fdf0691d"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.905718 13386 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal/instance:
uuid: "632ba412a681447e93af32d7fdf0691d"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.910508 13386 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20250624 07:55:07.913936 13403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:07.915297 13386 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250624 07:55:07.915448 13386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
uuid: "632ba412a681447e93af32d7fdf0691d"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:07.915632 13386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:07.944397 13385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 4
W20250624 07:55:07.946377 13385 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:07.948662 13386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:07.949929 13386 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:07.950136 13386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:07.954756 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 2
W20250624 07:55:07.956594 13315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:07.970466 13386 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.60:33449
I20250624 07:55:07.970484 13454 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.60:33449 every 8 connection(s)
I20250624 07:55:07.971855 13386 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
I20250624 07:55:07.975893 13455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:07.977319 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 13386
I20250624 07:55:07.977483 13008 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal/instance
I20250624 07:55:07.981472 13455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } has no permanent_uuid. Determining permanent_uuid...
I20250624 07:55:07.994975 13455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } has no permanent_uuid. Determining permanent_uuid...
I20250624 07:55:08.002135 13455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } has no permanent_uuid. Determining permanent_uuid...
I20250624 07:55:08.014904 13455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap starting.
I20250624 07:55:08.019049 13455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:08.020455 13455 log.cc:826] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:08.022874 13455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: No bootstrap required, opened a new log
I20250624 07:55:08.027779 13455 raft_consensus.cc:357] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.028044 13455 raft_consensus.cc:383] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:08.028147 13455 raft_consensus.cc:738] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Initialized, Role: FOLLOWER
I20250624 07:55:08.028702 13455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.029524 13463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.029755 13463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:08.030471 13455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:08.030755 13463 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:08.030931 13463 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.032912 13463 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
I20250624 07:55:08.041319 13386 master_runner.cc:186] Error getting master registration for 127.12.180.62:44637: OK, instance_id { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" instance_seqno: 1750751707131413 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250624 07:55:08.047751 13386 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:55:08.048341 13477 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250624 07:55:08.048445 13477 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250624 07:55:08.062736 13315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 3
I20250624 07:55:08.080770 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap starting.
I20250624 07:55:08.085103 13315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:08.086824 13315 log.cc:826] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:08.092386 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: No bootstrap required, opened a new log
I20250624 07:55:08.100428 13315 raft_consensus.cc:357] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.100800 13315 raft_consensus.cc:383] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:08.100915 13315 raft_consensus.cc:738] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588e956d4ac040deb3c06e57bd3c6b88, State: Initialized, Role: FOLLOWER
I20250624 07:55:08.101673 13315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.102742 13481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.103062 13481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:08.104131 13315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:08.104566 13290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
I20250624 07:55:08.105649 13290 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 0.
I20250624 07:55:08.106398 13407 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters:
I20250624 07:55:08.106791 13463 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:08.106951 13463 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:08.107052 13463 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:08.110421 13463 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.111586 13290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88"
I20250624 07:55:08.111775 13290 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:08.115281 13290 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 1.
I20250624 07:55:08.115794 13407 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters:
I20250624 07:55:08.117070 13484 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:08.117812 13463 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 1 election: Requested vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:08.120463 13269 tablet.cc:2378] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:55:08.123067 13484 raft_consensus.cc:695] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 1 LEADER]: Becoming Leader. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Running, Role: LEADER
I20250624 07:55:08.123950 13484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.142518 13463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 1 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.142822 13463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:08.150736 13493 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:08.152930 13493 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:08.169804 13290 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Refusing update from remote peer 632ba412a681447e93af32d7fdf0691d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 07:55:08.170886 13463 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Connected to new peer: Peer: permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:08.175747 13249 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: OK, instance_id { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" instance_seqno: 1750751707548209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250624 07:55:08.185022 13249 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:55:08.194828 13501 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250624 07:55:08.194972 13501 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250624 07:55:08.204650 13409 tablet.cc:2378] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:55:08.207034 13481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 1 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.207327 13481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:08.226578 13481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.226809 13481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:08.235291 13463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.235563 13463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:08.238235 13493 catalog_manager.cc:1349] Generated new cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:08.238355 13493 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:08.245580 13385 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } attempt: 5
I20250624 07:55:08.255052 13493 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 07:55:08.256786 13493 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:08.261866 13385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Bootstrap starting.
I20250624 07:55:08.265857 13385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:08.267161 13385 log.cc:826] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:08.282109 13385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: No bootstrap required, opened a new log
I20250624 07:55:08.282141 13493 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Generated new TSK 0
I20250624 07:55:08.283968 13493 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:08.286420 13493 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:08.289300 13493 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250624 07:55:08.289631 13385 raft_consensus.cc:357] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.290061 13385 raft_consensus.cc:383] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:08.290194 13385 raft_consensus.cc:738] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52afd679ffb241a3a9022cc850c03493, State: Initialized, Role: FOLLOWER
I20250624 07:55:08.290935 13385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:08.303207 13508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.304456 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.180.1:0
--local_ip_for_outbound_sockets=127.12.180.1
--webserver_interface=127.12.180.1
--webserver_port=0
--tserver_master_addrs=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250624 07:55:08.305752 13508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:08.305548 13385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:08.305971 13358 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:08.306959 13359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493"
I20250624 07:55:08.307171 13360 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493" is_pre_election: true
I20250624 07:55:08.311729 13358 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 1 FOLLOWER]: Refusing update from remote peer 632ba412a681447e93af32d7fdf0691d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 07:55:08.312693 13484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Connected to new peer: Peer: permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:08.353263 13508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 1 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.353628 13508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:08.354259 13516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.354759 13516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:08.377003 13529 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250624 07:55:08.763608 13484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:08.763943 13484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
W20250624 07:55:08.763885 13512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:08.764971 13512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:08.765161 13512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:08.772840 13512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:08.773064 13512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.180.1
I20250624 07:55:08.782967 13512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.180.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.180.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.180.1
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:08.784626 13512 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:08.786525 13512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:08.798266 13538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:08.798202 13536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:08.798235 13535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:08.799317 13512 server_base.cc:1048] running on GCE node
I20250624 07:55:08.799880 13512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:08.800843 13512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:08.802032 13512 hybrid_clock.cc:648] HybridClock initialized: now 1750751708801988 us; error 41 us; skew 500 ppm
I20250624 07:55:08.802371 13512 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:08.804452 13512 webserver.cc:469] Webserver started at http://127.12.180.1:37575/ using document root <none> and password file <none>
I20250624 07:55:08.805202 13512 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:08.805326 13512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:08.805661 13512 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:55:08.809407 13512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data/instance:
uuid: "dbb2615829f748ef929761ed90057cb7"
format_stamp: "Formatted at 2025-06-24 07:55:08 on dist-test-slave-68nn"
I20250624 07:55:08.810204 13512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/instance:
uuid: "dbb2615829f748ef929761ed90057cb7"
format_stamp: "Formatted at 2025-06-24 07:55:08 on dist-test-slave-68nn"
I20250624 07:55:08.815128 13512 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250624 07:55:08.819159 13544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:08.821403 13512 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250624 07:55:08.821583 13512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal
uuid: "dbb2615829f748ef929761ed90057cb7"
format_stamp: "Formatted at 2025-06-24 07:55:08 on dist-test-slave-68nn"
I20250624 07:55:08.821807 13512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:08.850307 13512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:08.851120 13512 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:08.851346 13512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:08.852274 13512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:55:08.853709 13512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:55:08.853786 13512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:08.853870 13512 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:55:08.853940 13512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:08.884318 13512 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.1:39271
I20250624 07:55:08.884356 13656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.1:39271 every 8 connection(s)
I20250624 07:55:08.885659 13512 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/data/info.pb
I20250624 07:55:08.892690 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 13512
I20250624 07:55:08.892952 13008 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/instance
I20250624 07:55:08.909211 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.180.2:0
--local_ip_for_outbound_sockets=127.12.180.2
--webserver_interface=127.12.180.2
--webserver_port=0
--tserver_master_addrs=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250624 07:55:08.914608 13659 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:08.914844 13658 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:08.915161 13658 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:08.915246 13657 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:08.915175 13659 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:08.915383 13657 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:08.916611 13658 heartbeater.cc:507] Master 127.12.180.62:44637 requested a full tablet report, sending...
I20250624 07:55:08.916651 13657 heartbeater.cc:507] Master 127.12.180.60:33449 requested a full tablet report, sending...
I20250624 07:55:08.916674 13659 heartbeater.cc:507] Master 127.12.180.61:46565 requested a full tablet report, sending...
I20250624 07:55:08.919076 13420 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:08.919076 13279 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:08.919171 13350 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:08.920680 13420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.180.1:33613
I20250624 07:55:09.197441 13501 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:09.197587 13501 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: loading cluster ID for follower catalog manager: success
I20250624 07:55:09.201295 13501 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: acquiring CA information for follower catalog manager: success
I20250624 07:55:09.203965 13501 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250624 07:55:09.249533 13667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:09.249843 13667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:09.249935 13667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:09.257998 13667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:09.258193 13667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.180.2
I20250624 07:55:09.267902 13667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.180.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.180.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.180.2
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:09.269389 13667 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:09.271147 13667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:09.282816 13672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:09.282886 13673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:09.282913 13675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:09.284346 13667 server_base.cc:1048] running on GCE node
I20250624 07:55:09.284888 13667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:09.285820 13667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:09.287009 13667 hybrid_clock.cc:648] HybridClock initialized: now 1750751709286961 us; error 50 us; skew 500 ppm
I20250624 07:55:09.287295 13667 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:09.289455 13667 webserver.cc:469] Webserver started at http://127.12.180.2:35557/ using document root <none> and password file <none>
I20250624 07:55:09.290148 13667 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:09.290289 13667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:09.290601 13667 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:55:09.292896 13667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data/instance:
uuid: "ab43ace4a8b44b908d247cad235e06df"
format_stamp: "Formatted at 2025-06-24 07:55:09 on dist-test-slave-68nn"
I20250624 07:55:09.293661 13667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/instance:
uuid: "ab43ace4a8b44b908d247cad235e06df"
format_stamp: "Formatted at 2025-06-24 07:55:09 on dist-test-slave-68nn"
I20250624 07:55:09.298765 13667 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20250624 07:55:09.301891 13681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:09.303170 13667 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250624 07:55:09.303323 13667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal
uuid: "ab43ace4a8b44b908d247cad235e06df"
format_stamp: "Formatted at 2025-06-24 07:55:09 on dist-test-slave-68nn"
I20250624 07:55:09.303504 13667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:09.328466 13667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:09.329349 13667 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:09.329593 13667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:09.330615 13667 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:55:09.332288 13667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:55:09.332383 13667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:09.332957 13667 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:55:09.333034 13667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250624 07:55:09.369267 13667 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.2:43137
I20250624 07:55:09.369328 13793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.2:43137 every 8 connection(s)
I20250624 07:55:09.370756 13667 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/data/info.pb
I20250624 07:55:09.371050 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 13667
I20250624 07:55:09.371245 13008 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/instance
I20250624 07:55:09.377148 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.180.3:0
--local_ip_for_outbound_sockets=127.12.180.3
--webserver_interface=127.12.180.3
--webserver_port=0
--tserver_master_addrs=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250624 07:55:09.386987 13529 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:09.387136 13529 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: loading cluster ID for follower catalog manager: success
I20250624 07:55:09.391113 13529 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: acquiring CA information for follower catalog manager: success
I20250624 07:55:09.393575 13529 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:09.406612 13794 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:09.407012 13794 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:09.408145 13794 heartbeater.cc:507] Master 127.12.180.60:33449 requested a full tablet report, sending...
I20250624 07:55:09.408702 13796 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:09.408852 13796 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:09.409257 13795 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:09.409395 13795 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:09.409397 13796 heartbeater.cc:507] Master 127.12.180.61:46565 requested a full tablet report, sending...
I20250624 07:55:09.409870 13420 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:09.409899 13795 heartbeater.cc:507] Master 127.12.180.62:44637 requested a full tablet report, sending...
I20250624 07:55:09.410722 13350 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:09.410878 13420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.180.2:38047
I20250624 07:55:09.411970 13279 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:09.423086 13657 heartbeater.cc:499] Master 127.12.180.60:33449 was elected leader, sending a full tablet report...
I20250624 07:55:09.705214 13803 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:09.705447 13803 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:09.707363 13803 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:09.707777 13358 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493" is_pre_election: true
I20250624 07:55:09.708539 13430 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "632ba412a681447e93af32d7fdf0691d" is_pre_election: true
I20250624 07:55:09.709333 13264 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:09.709759 13803 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250624 07:55:09.734289 13802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:09.734643 13802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:09.734786 13802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:09.743057 13802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:09.743275 13802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.180.3
I20250624 07:55:09.752390 13802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.180.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.180.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.180.3
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:09.754082 13802 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:09.755996 13802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:09.767812 13811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:09.767874 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:09.767838 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:09.768263 13802 server_base.cc:1048] running on GCE node
I20250624 07:55:09.768906 13802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:09.769764 13802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:09.770951 13802 hybrid_clock.cc:648] HybridClock initialized: now 1750751709770894 us; error 43 us; skew 500 ppm
I20250624 07:55:09.771263 13802 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:09.773382 13802 webserver.cc:469] Webserver started at http://127.12.180.3:38113/ using document root <none> and password file <none>
I20250624 07:55:09.774070 13802 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:09.774176 13802 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:09.774516 13802 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 07:55:09.777522 13802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data/instance:
uuid: "f96059407ad14a70971e622891d2f99c"
format_stamp: "Formatted at 2025-06-24 07:55:09 on dist-test-slave-68nn"
I20250624 07:55:09.778329 13802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/instance:
uuid: "f96059407ad14a70971e622891d2f99c"
format_stamp: "Formatted at 2025-06-24 07:55:09 on dist-test-slave-68nn"
I20250624 07:55:09.783149 13802 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250624 07:55:09.786247 13817 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:09.787418 13802 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250624 07:55:09.787564 13802 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal
uuid: "f96059407ad14a70971e622891d2f99c"
format_stamp: "Formatted at 2025-06-24 07:55:09 on dist-test-slave-68nn"
I20250624 07:55:09.787766 13802 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:09.799101 13802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:09.799883 13802 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:09.800103 13802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:09.801470 13802 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 07:55:09.803022 13802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 07:55:09.803110 13802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:09.803210 13802 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 07:55:09.803275 13802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:09.838439 13802 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.3:33179
I20250624 07:55:09.838511 13929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.3:33179 every 8 connection(s)
I20250624 07:55:09.839802 13802 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/data/info.pb
I20250624 07:55:09.847364 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 13802
I20250624 07:55:09.847560 13008 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/instance
I20250624 07:55:09.857623 13932 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:09.857648 13930 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:09.857777 13931 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:09.858094 13931 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:09.858098 13932 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:09.858100 13930 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:09.859189 13931 heartbeater.cc:507] Master 127.12.180.62:44637 requested a full tablet report, sending...
I20250624 07:55:09.859229 13930 heartbeater.cc:507] Master 127.12.180.60:33449 requested a full tablet report, sending...
I20250624 07:55:09.859251 13932 heartbeater.cc:507] Master 127.12.180.61:46565 requested a full tablet report, sending...
I20250624 07:55:09.860702 13350 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:09.860802 13279 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:09.860797 13420 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:09.861757 13420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.180.3:57373
I20250624 07:55:09.871754 13008 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 07:55:09.902901 13420 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_e87f394359f24830b4171bf6b60d4ab2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.902973 13419 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_8a84bf94dbd94e65946d5e56b69172be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.903169 13416 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_ba78959886d2470285dacaf16007267d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.903839 13418 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_0047240c5fdf4b85bc2b5fda6bdcef32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.904521 13417 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_6e36abec7aef4670b05e620d91bba579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.905076 13415 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_953a2fe7f1694bb7b1afcbbaddd632b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.905475 13414 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.908277 13413 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_2974357596ca4893b46d7199b1367e2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 07:55:09.911276 13420 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e87f394359f24830b4171bf6b60d4ab2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:55:09.911713 13412 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_02926d7fe80d4545a80eb082aa7d2671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.912941 13411 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42490:
name: "default.Table_fc2c06ca3ddc4d78baaced0d6345b281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:09.918339 13794 heartbeater.cc:499] Master 127.12.180.60:33449 was elected leader, sending a full tablet report...
I20250624 07:55:09.961819 13008 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu pid 13386
I20250624 07:55:09.962903 13450 generic_service.cc:196] Checking for leaks (request via RPC)
I20250624 07:55:10.106638 13008 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu with pid 13386
W20250624 07:55:10.144817 13937 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.180.60:33449)
I20250624 07:55:10.146647 13977 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:10.146939 13977 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:10.149053 13290 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
I20250624 07:55:10.149966 13977 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:10.151455 13794 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.12.180.60:33449 (error 108)
W20250624 07:55:10.152104 13657 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.12.180.60:33449 (error 108)
W20250624 07:55:10.152247 13334 connection.cc:537] client connection to 127.12.180.60:33449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:10.169243 13979 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:10.153369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.180.60:33449 (local address 127.0.0.1:42512)
0624 07:55:10.153883 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0624 07:55:10.153904 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0624 07:55:10.154087 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0624 07:55:10.168163 (+ 14076us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
W20250624 07:55:10.169621 13941 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:10.146568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.180.60:33449 (local address 127.0.0.1:42500)
0624 07:55:10.149778 (+ 3210us) negotiation.cc:107] Waiting for socket to connect
0624 07:55:10.149799 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0624 07:55:10.149977 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0624 07:55:10.168256 (+ 18279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2956}
W20250624 07:55:10.173008 13334 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 07:55:10.173223 13334 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493; no voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:10.173652 13977 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250624 07:55:10.241330 13980 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:10.241504 13980 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:10.242108 13980 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:10.242658 13358 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493" is_pre_election: true
I20250624 07:55:10.242966 13358 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588e956d4ac040deb3c06e57bd3c6b88 in term 1.
I20250624 07:55:10.243438 13266 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters:
I20250624 07:55:10.243762 13980 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 07:55:10.243877 13980 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:10.243973 13980 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 1 FOLLOWER]: Advancing to term 2
W20250624 07:55:10.244478 13264 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:10.246798 13980 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:10.247433 13980 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 election: Requested vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:10.247850 13358 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493"
I20250624 07:55:10.248101 13358 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 1 FOLLOWER]: Advancing to term 2
W20250624 07:55:10.249042 13264 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:10.251057 13358 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588e956d4ac040deb3c06e57bd3c6b88 in term 2.
I20250624 07:55:10.251629 13266 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters: 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:10.252115 13980 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:55:10.252959 13980 raft_consensus.cc:695] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 LEADER]: Becoming Leader. State: Replica: 588e956d4ac040deb3c06e57bd3c6b88, State: Running, Role: LEADER
I20250624 07:55:10.253505 13980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:10.255846 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588e956d4ac040deb3c06e57bd3c6b88. Latest consensus state: current_term: 2 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:10.256127 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:10.256992 13985 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:10.260150 13985 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:10.261368 13985 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:10.261458 13985 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:10.263099 13985 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:10.264147 13985 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Loaded TSK: 0
I20250624 07:55:10.265399 13985 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:10.266503 13985 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:10.267426 13985 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250624 07:55:10.280792 13279 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_fc2c06ca3ddc4d78baaced0d6345b281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.283107 13280 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_02926d7fe80d4545a80eb082aa7d2671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.283807 13278 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_2974357596ca4893b46d7199b1367e2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.284484 13277 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.285248 13276 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_8a84bf94dbd94e65946d5e56b69172be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.292168 13275 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_e87f394359f24830b4171bf6b60d4ab2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.296919 13274 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_0047240c5fdf4b85bc2b5fda6bdcef32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.297013 13273 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_6e36abec7aef4670b05e620d91bba579"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.297672 13272 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_ba78959886d2470285dacaf16007267d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.297792 13271 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_953a2fe7f1694bb7b1afcbbaddd632b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 07:55:10.303632 13279 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fc2c06ca3ddc4d78baaced0d6345b281 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:55:10.363718 13930 heartbeater.cc:499] Master 127.12.180.60:33449 was elected leader, sending a full tablet report...
W20250624 07:55:10.364979 13930 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:10.447894 13279 catalog_manager.cc:2149] created HMS catalog entry for table [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:10.452425 13358 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 2 FOLLOWER]: Refusing update from remote peer 588e956d4ac040deb3c06e57bd3c6b88: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20250624 07:55:10.453065 13264 consensus_peers.cc:487] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 -> Peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Couldn't send request to peer 632ba412a681447e93af32d7fdf0691d. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:55:10.453150 13983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250624 07:55:10.454936 13274 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.455286 13278 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2974357596ca4893b46d7199b1367e2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.456234 13272 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ba78959886d2470285dacaf16007267d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.456502 13276 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8a84bf94dbd94e65946d5e56b69172be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.456235 13277 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b66cb6b1a255496bb4909de6a39e80fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.457697 13273 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6e36abec7aef4670b05e620d91bba579 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.457418 13275 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e87f394359f24830b4171bf6b60d4ab2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.458720 13271 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250624 07:55:10.454939 13280 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_02926d7fe80d4545a80eb082aa7d2671 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 07:55:10.472949 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588e956d4ac040deb3c06e57bd3c6b88. Latest consensus state: current_term: 2 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:10.473212 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:10.476063 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:10.476320 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:10.479676 13279 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48636:
table { table_name: "default.Table_fc2c06ca3ddc4d78baaced0d6345b281" } new_table_name: "default.Table_4b83c91d0994483c8da55d71dafb8eeb" modify_external_catalogs: true
I20250624 07:55:10.486538 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:10.486774 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:10.497308 13274 catalog_manager.cc:2149] created HMS catalog entry for table [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:10.509866 13592 tablet_service.cc:1468] Processing CreateTablet for tablet 325d48f2c20143129d28d0daacc0908a (DEFAULT_TABLE table=default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.516222 13592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325d48f2c20143129d28d0daacc0908a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.516489 13274 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48636:
table { table_name: "default.Table_0047240c5fdf4b85bc2b5fda6bdcef32" } new_table_name: "default.Table_34f740d34f55471b82a8fd8be1423bbf" modify_external_catalogs: true
I20250624 07:55:10.517009 13865 tablet_service.cc:1468] Processing CreateTablet for tablet 325d48f2c20143129d28d0daacc0908a (DEFAULT_TABLE table=default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.518393 13865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325d48f2c20143129d28d0daacc0908a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.522346 13591 tablet_service.cc:1468] Processing CreateTablet for tablet c8ef7b6a115e4985a123b2792b5360c0 (DEFAULT_TABLE table=default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.522873 13591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ef7b6a115e4985a123b2792b5360c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.528841 13864 tablet_service.cc:1468] Processing CreateTablet for tablet c8ef7b6a115e4985a123b2792b5360c0 (DEFAULT_TABLE table=default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.529472 13864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ef7b6a115e4985a123b2792b5360c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.530303 13278 catalog_manager.cc:2149] created HMS catalog entry for table [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:10.541054 13729 tablet_service.cc:1468] Processing CreateTablet for tablet 325d48f2c20143129d28d0daacc0908a (DEFAULT_TABLE table=default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.543622 13728 tablet_service.cc:1468] Processing CreateTablet for tablet c8ef7b6a115e4985a123b2792b5360c0 (DEFAULT_TABLE table=default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.544198 13278 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48636:
table { table_name: "default.Table_2974357596ca4893b46d7199b1367e2f" } new_table_name: "default.Table_77a9deb86184406588f30ae47cdb52d6" modify_external_catalogs: true
I20250624 07:55:10.544440 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ef7b6a115e4985a123b2792b5360c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.550448 13729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325d48f2c20143129d28d0daacc0908a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.567561 13590 tablet_service.cc:1468] Processing CreateTablet for tablet c95dc3a9620745c6a7020ca7434e2c1e (DEFAULT_TABLE table=default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.568207 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95dc3a9620745c6a7020ca7434e2c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.568370 13997 tablet_bootstrap.cc:492] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.570472 13863 tablet_service.cc:1468] Processing CreateTablet for tablet c95dc3a9620745c6a7020ca7434e2c1e (DEFAULT_TABLE table=default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.571029 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95dc3a9620745c6a7020ca7434e2c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.580017 13998 tablet_bootstrap.cc:492] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:10.580060 13728 tablet_service.cc:1468] Processing CreateTablet for tablet c95dc3a9620745c6a7020ca7434e2c1e (DEFAULT_TABLE table=default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.581375 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95dc3a9620745c6a7020ca7434e2c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.582870 13999 tablet_bootstrap.cc:492] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.583753 13998 tablet_bootstrap.cc:654] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.589651 13999 tablet_bootstrap.cc:654] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.591998 13998 log.cc:826] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:10.593887 13999 log.cc:826] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:10.598628 13999 tablet_bootstrap.cc:492] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.598927 13999 ts_tablet_manager.cc:1397] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20250624 07:55:10.603389 13997 tablet_bootstrap.cc:654] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.604739 13997 log.cc:826] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:10.605353 13999 raft_consensus.cc:357] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } }
I20250624 07:55:10.605857 13999 raft_consensus.cc:383] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.606045 13999 raft_consensus.cc:738] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.607016 13999 consensus_queue.cc:260] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } }
I20250624 07:55:10.612569 13998 tablet_bootstrap.cc:492] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.612880 13998 ts_tablet_manager.cc:1397] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.000s
I20250624 07:55:10.614171 13997 tablet_bootstrap.cc:492] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.614503 13997 ts_tablet_manager.cc:1397] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.000s
I20250624 07:55:10.617761 13998 raft_consensus.cc:357] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } }
I20250624 07:55:10.618046 13998 raft_consensus.cc:383] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.618155 13998 raft_consensus.cc:738] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.619751 13999 ts_tablet_manager.cc:1428] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20250624 07:55:10.620178 13795 heartbeater.cc:499] Master 127.12.180.62:44637 was elected leader, sending a full tablet report...
I20250624 07:55:10.620622 13998 consensus_queue.cc:260] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } }
I20250624 07:55:10.621644 13999 tablet_bootstrap.cc:492] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.624115 13999 tablet_bootstrap.cc:654] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
W20250624 07:55:10.638513 13800 tablet.cc:2378] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:55:10.640667 13997 raft_consensus.cc:357] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } }
I20250624 07:55:10.641178 13997 raft_consensus.cc:383] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.641291 13997 raft_consensus.cc:738] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.641856 13272 catalog_manager.cc:2149] created HMS catalog entry for table [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:10.642895 13997 consensus_queue.cc:260] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } }
I20250624 07:55:10.644274 13931 heartbeater.cc:499] Master 127.12.180.62:44637 was elected leader, sending a full tablet report...
I20250624 07:55:10.647203 13999 tablet_bootstrap.cc:492] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.647421 13999 ts_tablet_manager.cc:1397] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.018s
I20250624 07:55:10.650347 13999 raft_consensus.cc:357] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.650607 13999 raft_consensus.cc:383] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.650724 13998 ts_tablet_manager.cc:1428] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.038s user 0.013s sys 0.016s
I20250624 07:55:10.650738 13999 raft_consensus.cc:738] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.650998 13999 consensus_queue.cc:260] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.651482 13999 ts_tablet_manager.cc:1428] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250624 07:55:10.651993 13999 tablet_bootstrap.cc:492] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.654829 13999 tablet_bootstrap.cc:654] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.658340 13997 ts_tablet_manager.cc:1428] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.044s user 0.015s sys 0.000s
I20250624 07:55:10.658624 13658 heartbeater.cc:499] Master 127.12.180.62:44637 was elected leader, sending a full tablet report...
I20250624 07:55:10.660449 13998 tablet_bootstrap.cc:492] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:10.661105 13997 tablet_bootstrap.cc:492] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.661968 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 75bf5e310d42413ca5da83762bf91e9d (DEFAULT_TABLE table=default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.662202 13590 tablet_service.cc:1468] Processing CreateTablet for tablet 75bf5e310d42413ca5da83762bf91e9d (DEFAULT_TABLE table=default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.662366 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75bf5e310d42413ca5da83762bf91e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.662585 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75bf5e310d42413ca5da83762bf91e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.663362 13997 tablet_bootstrap.cc:654] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.662189 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 75bf5e310d42413ca5da83762bf91e9d (DEFAULT_TABLE table=default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.665706 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75bf5e310d42413ca5da83762bf91e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.670115 13997 tablet_bootstrap.cc:492] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.670370 13997 ts_tablet_manager.cc:1397] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20250624 07:55:10.671327 13997 raft_consensus.cc:357] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.671607 13997 raft_consensus.cc:383] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.671751 13997 raft_consensus.cc:738] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.672066 13997 consensus_queue.cc:260] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.672646 13997 ts_tablet_manager.cc:1428] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250624 07:55:10.672979 13997 tablet_bootstrap.cc:492] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.674831 13997 tablet_bootstrap.cc:654] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.677877 13998 tablet_bootstrap.cc:654] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.685945 13272 master_service.cc:948] ReplaceTablet: received request to replace tablet c8ef7b6a115e4985a123b2792b5360c0 from {username='slave'} at 127.0.0.1:48636
I20250624 07:55:10.686558 13999 tablet_bootstrap.cc:492] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.686735 13999 ts_tablet_manager.cc:1397] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.006s
I20250624 07:55:10.686746 13998 tablet_bootstrap.cc:492] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.686915 13998 ts_tablet_manager.cc:1397] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20250624 07:55:10.687513 13999 raft_consensus.cc:357] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.687681 13999 raft_consensus.cc:383] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.687669 13998 raft_consensus.cc:357] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.687759 13999 raft_consensus.cc:738] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.687836 13998 raft_consensus.cc:383] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.687908 13998 raft_consensus.cc:738] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.688026 13999 consensus_queue.cc:260] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.688154 13998 consensus_queue.cc:260] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.688591 13998 ts_tablet_manager.cc:1428] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250624 07:55:10.690506 13998 tablet_bootstrap.cc:492] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:10.691102 13999 ts_tablet_manager.cc:1428] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250624 07:55:10.691357 13999 tablet_bootstrap.cc:492] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.692592 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--webserver_interface=127.12.180.60
--webserver_port=42595
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250624 07:55:10.693204 13999 tablet_bootstrap.cc:654] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.698328 13272 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Sending DeleteTablet for 3 replicas of tablet c8ef7b6a115e4985a123b2792b5360c0
I20250624 07:55:10.699471 13590 tablet_service.cc:1515] Processing DeleteTablet for tablet c8ef7b6a115e4985a123b2792b5360c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 31617090f65045f08d94b793ff525275) from {username='slave'} at 127.0.0.1:37434
I20250624 07:55:10.700901 13998 tablet_bootstrap.cc:654] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.703243 13728 tablet_service.cc:1515] Processing DeleteTablet for tablet c8ef7b6a115e4985a123b2792b5360c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 31617090f65045f08d94b793ff525275) from {username='slave'} at 127.0.0.1:34354
I20250624 07:55:10.704253 14017 tablet_replica.cc:331] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:10.704805 13863 tablet_service.cc:1515] Processing DeleteTablet for tablet c8ef7b6a115e4985a123b2792b5360c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 31617090f65045f08d94b793ff525275) from {username='slave'} at 127.0.0.1:44562
I20250624 07:55:10.707538 13272 catalog_manager.cc:6652] ReplaceTablet: tablet c8ef7b6a115e4985a123b2792b5360c0 deleted and replaced by tablet 31617090f65045f08d94b793ff525275
I20250624 07:55:10.716310 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 31617090f65045f08d94b793ff525275 (DEFAULT_TABLE table=default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.716583 13591 tablet_service.cc:1468] Processing CreateTablet for tablet 31617090f65045f08d94b793ff525275 (DEFAULT_TABLE table=default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.716778 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31617090f65045f08d94b793ff525275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.716799 13865 tablet_service.cc:1468] Processing CreateTablet for tablet 31617090f65045f08d94b793ff525275 (DEFAULT_TABLE table=default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.716979 13591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31617090f65045f08d94b793ff525275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.717168 13865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31617090f65045f08d94b793ff525275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.722435 14003 raft_consensus.cc:491] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:10.722685 14003 raft_consensus.cc:513] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.725068 14003 leader_election.cc:290] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:10.733215 13272 master_service.cc:948] ReplaceTablet: received request to replace tablet 325d48f2c20143129d28d0daacc0908a from {username='slave'} at 127.0.0.1:48636
I20250624 07:55:10.735529 14017 raft_consensus.cc:2241] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:10.736291 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8ef7b6a115e4985a123b2792b5360c0" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:10.737529 13997 tablet_bootstrap.cc:492] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.738055 13997 ts_tablet_manager.cc:1397] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.009s
I20250624 07:55:10.739068 13997 raft_consensus.cc:357] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.739552 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8ef7b6a115e4985a123b2792b5360c0" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:10.740160 13997 raft_consensus.cc:383] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.740176 13885 raft_consensus.cc:2466] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:10.740268 13997 raft_consensus.cc:738] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.740800 13997 consensus_queue.cc:260] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.741357 13997 ts_tablet_manager.cc:1428] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250624 07:55:10.741649 13997 tablet_bootstrap.cc:492] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.744575 13272 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Sending DeleteTablet for 3 replicas of tablet 325d48f2c20143129d28d0daacc0908a
I20250624 07:55:10.740919 13683 leader_election.cc:304] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:10.745590 14018 tablet_replica.cc:331] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:10.745754 14003 raft_consensus.cc:2704] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20250624 07:55:10.746284 14018 raft_consensus.cc:2241] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:10.746320 14017 raft_consensus.cc:2270] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:10.746719 14018 raft_consensus.cc:2270] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:10.736281 14015 tablet_replica.cc:331] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:10.751183 13728 tablet_service.cc:1515] Processing DeleteTablet for tablet 325d48f2c20143129d28d0daacc0908a with delete_type TABLET_DATA_DELETED (replaced by tablet 09b765ab702f4817879ab30ebb558336) from {username='slave'} at 127.0.0.1:34354
I20250624 07:55:10.750427 13863 tablet_service.cc:1515] Processing DeleteTablet for tablet 325d48f2c20143129d28d0daacc0908a with delete_type TABLET_DATA_DELETED (replaced by tablet 09b765ab702f4817879ab30ebb558336) from {username='slave'} at 127.0.0.1:44562
I20250624 07:55:10.752614 14017 ts_tablet_manager.cc:1905] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:10.749325 13612 raft_consensus.cc:2466] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:10.751971 13590 tablet_service.cc:1515] Processing DeleteTablet for tablet 325d48f2c20143129d28d0daacc0908a with delete_type TABLET_DATA_DELETED (replaced by tablet 09b765ab702f4817879ab30ebb558336) from {username='slave'} at 127.0.0.1:37434
I20250624 07:55:10.747819 13997 tablet_bootstrap.cc:654] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.755430 14018 ts_tablet_manager.cc:1905] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:10.757803 13272 catalog_manager.cc:6652] ReplaceTablet: tablet 325d48f2c20143129d28d0daacc0908a deleted and replaced by tablet 09b765ab702f4817879ab30ebb558336
I20250624 07:55:10.759449 13999 tablet_bootstrap.cc:492] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.759627 13999 ts_tablet_manager.cc:1397] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.068s user 0.018s sys 0.028s
I20250624 07:55:10.760393 13999 raft_consensus.cc:357] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.760593 13999 raft_consensus.cc:383] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.761008 13999 raft_consensus.cc:738] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.761276 13999 consensus_queue.cc:260] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.762977 13590 tablet_service.cc:1468] Processing CreateTablet for tablet 09b765ab702f4817879ab30ebb558336 (DEFAULT_TABLE table=default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.763410 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b765ab702f4817879ab30ebb558336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.765575 13999 ts_tablet_manager.cc:1428] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250624 07:55:10.766291 14015 raft_consensus.cc:2241] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:10.767129 14015 raft_consensus.cc:2270] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:10.768708 13999 tablet_bootstrap.cc:492] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.770241 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 09b765ab702f4817879ab30ebb558336 (DEFAULT_TABLE table=default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.770435 13999 tablet_bootstrap.cc:654] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.770691 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b765ab702f4817879ab30ebb558336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.771631 14018 ts_tablet_manager.cc:1918] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250624 07:55:10.771806 14018 log.cc:1199] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/c8ef7b6a115e4985a123b2792b5360c0
I20250624 07:55:10.772205 14018 ts_tablet_manager.cc:1939] T c8ef7b6a115e4985a123b2792b5360c0 P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:10.774416 13265 catalog_manager.cc:4928] TS f96059407ad14a70971e622891d2f99c (127.12.180.3:33179): tablet c8ef7b6a115e4985a123b2792b5360c0 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]) successfully deleted
I20250624 07:55:10.777282 14018 tablet_replica.cc:331] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:10.777659 14018 raft_consensus.cc:2241] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:10.778124 14018 raft_consensus.cc:2270] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:10.778388 13999 tablet_bootstrap.cc:492] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.778538 13999 ts_tablet_manager.cc:1397] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250624 07:55:10.779357 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 09b765ab702f4817879ab30ebb558336 (DEFAULT_TABLE table=default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.779353 13999 raft_consensus.cc:357] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.779534 13999 raft_consensus.cc:383] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.779625 13999 raft_consensus.cc:738] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.779747 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b765ab702f4817879ab30ebb558336. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.779821 13999 consensus_queue.cc:260] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.783298 13999 ts_tablet_manager.cc:1428] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250624 07:55:10.784037 14015 ts_tablet_manager.cc:1905] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:10.786152 13999 tablet_bootstrap.cc:492] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.787240 13997 tablet_bootstrap.cc:492] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.787446 13997 ts_tablet_manager.cc:1397] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.009s
I20250624 07:55:10.787863 13999 tablet_bootstrap.cc:654] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.788376 13997 raft_consensus.cc:357] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.788736 13997 raft_consensus.cc:383] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.788874 13997 raft_consensus.cc:738] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.789237 13997 consensus_queue.cc:260] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.793150 13276 catalog_manager.cc:2149] created HMS catalog entry for table [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:10.793179 14003 raft_consensus.cc:491] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:10.793330 14003 raft_consensus.cc:513] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.793957 14003 leader_election.cc:290] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:10.794456 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c95dc3a9620745c6a7020ca7434e2c1e" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:10.794907 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c95dc3a9620745c6a7020ca7434e2c1e" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:10.795099 13612 raft_consensus.cc:2466] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
W20250624 07:55:10.796005 13683 leader_election.cc:343] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f96059407ad14a70971e622891d2f99c (127.12.180.3:33179): Illegal state: must be running to vote when last-logged opid is not known
I20250624 07:55:10.796730 13272 master_service.cc:948] ReplaceTablet: received request to replace tablet c8ef7b6a115e4985a123b2792b5360c0 from {username='slave'} at 127.0.0.1:48636
I20250624 07:55:10.797230 13272 master_service.cc:948] ReplaceTablet: received request to replace tablet c8ef7b6a115e4985a123b2792b5360c0 from {username='slave'} at 127.0.0.1:48636
I20250624 07:55:10.797613 13272 master_service.cc:948] ReplaceTablet: received request to replace tablet c8ef7b6a115e4985a123b2792b5360c0 from {username='slave'} at 127.0.0.1:48636
I20250624 07:55:10.799038 13683 leader_election.cc:304] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters: f96059407ad14a70971e622891d2f99c
I20250624 07:55:10.799602 14003 raft_consensus.cc:2802] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:10.799747 14003 raft_consensus.cc:491] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:10.799849 14003 raft_consensus.cc:3058] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:10.801501 13276 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_26071aa0d9c8459b8f1d9583dedee21a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.802711 14003 raft_consensus.cc:513] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.803537 14003 leader_election.cc:290] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Requested vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:10.804250 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c95dc3a9620745c6a7020ca7434e2c1e" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:10.804478 13612 raft_consensus.cc:3058] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:10.804855 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c95dc3a9620745c6a7020ca7434e2c1e" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:10.805959 13998 tablet_bootstrap.cc:492] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.806100 13998 ts_tablet_manager.cc:1397] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.116s user 0.001s sys 0.008s
I20250624 07:55:10.806867 13998 raft_consensus.cc:357] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.807031 13998 raft_consensus.cc:383] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.807106 13998 raft_consensus.cc:738] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.807296 13998 consensus_queue.cc:260] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.807756 13998 ts_tablet_manager.cc:1428] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250624 07:55:10.807754 13612 raft_consensus.cc:2466] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:10.808013 13998 tablet_bootstrap.cc:492] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
W20250624 07:55:10.806362 13683 leader_election.cc:343] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f96059407ad14a70971e622891d2f99c (127.12.180.3:33179): Illegal state: must be running to vote when last-logged opid is not known
I20250624 07:55:10.810501 13683 leader_election.cc:304] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters: f96059407ad14a70971e622891d2f99c
I20250624 07:55:10.811034 14003 raft_consensus.cc:2802] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:10.811888 13997 ts_tablet_manager.cc:1428] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250624 07:55:10.813969 14018 ts_tablet_manager.cc:1905] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:10.814486 13998 tablet_bootstrap.cc:654] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.820067 13999 tablet_bootstrap.cc:492] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.820259 13999 ts_tablet_manager.cc:1397] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.003s
I20250624 07:55:10.821091 13999 raft_consensus.cc:357] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.821300 13999 raft_consensus.cc:383] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.821377 13999 raft_consensus.cc:738] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.821653 13999 consensus_queue.cc:260] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.823743 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 82f70c673a9749be9fc70112188be360 (DEFAULT_TABLE table=default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.824278 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f70c673a9749be9fc70112188be360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.798084 13272 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48636:
table { table_name: "default.Table_ba78959886d2470285dacaf16007267d" } new_table_name: "default.Table_f6122adb52c04c12b5cad442b855c049" modify_external_catalogs: true
I20250624 07:55:10.825168 14003 raft_consensus.cc:695] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Becoming Leader. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Running, Role: LEADER
I20250624 07:55:10.825807 14003 consensus_queue.cc:237] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.827689 14017 ts_tablet_manager.cc:1918] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250624 07:55:10.830392 13997 tablet_bootstrap.cc:492] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.832552 13997 tablet_bootstrap.cc:654] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.829088 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 82f70c673a9749be9fc70112188be360 (DEFAULT_TABLE table=default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.833180 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f70c673a9749be9fc70112188be360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.833298 14015 ts_tablet_manager.cc:1918] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250624 07:55:10.833499 14015 log.cc:1199] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/c8ef7b6a115e4985a123b2792b5360c0
I20250624 07:55:10.834031 14015 ts_tablet_manager.cc:1939] T c8ef7b6a115e4985a123b2792b5360c0 P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:10.834453 13590 tablet_service.cc:1468] Processing CreateTablet for tablet 82f70c673a9749be9fc70112188be360 (DEFAULT_TABLE table=default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.834960 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f70c673a9749be9fc70112188be360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.830838 13999 ts_tablet_manager.cc:1428] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250624 07:55:10.839249 14017 log.cc:1199] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/c8ef7b6a115e4985a123b2792b5360c0
I20250624 07:55:10.839716 14017 ts_tablet_manager.cc:1939] T c8ef7b6a115e4985a123b2792b5360c0 P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:10.841851 14018 ts_tablet_manager.cc:1918] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250624 07:55:10.841997 14018 log.cc:1199] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/325d48f2c20143129d28d0daacc0908a
I20250624 07:55:10.842418 13265 catalog_manager.cc:4928] TS dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271): tablet c8ef7b6a115e4985a123b2792b5360c0 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]) successfully deleted
W20250624 07:55:10.843248 13933 tablet.cc:2378] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:55:10.843576 14017 tablet_replica.cc:331] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:10.843789 14017 raft_consensus.cc:2241] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:10.843984 14017 raft_consensus.cc:2270] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:10.844285 13264 catalog_manager.cc:4928] TS ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): tablet c8ef7b6a115e4985a123b2792b5360c0 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]) successfully deleted
I20250624 07:55:10.844746 14015 tablet_replica.cc:331] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:10.844956 14015 raft_consensus.cc:2241] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:10.845139 14015 raft_consensus.cc:2270] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:10.845976 14015 ts_tablet_manager.cc:1905] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:10.846474 14018 ts_tablet_manager.cc:1939] T 325d48f2c20143129d28d0daacc0908a P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:10.847497 13265 catalog_manager.cc:4928] TS f96059407ad14a70971e622891d2f99c (127.12.180.3:33179): tablet 325d48f2c20143129d28d0daacc0908a (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]) successfully deleted
I20250624 07:55:10.847910 13999 tablet_bootstrap.cc:492] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.848048 13997 tablet_bootstrap.cc:492] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.848198 13997 ts_tablet_manager.cc:1397] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.004s
I20250624 07:55:10.848932 13997 raft_consensus.cc:357] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.849138 13997 raft_consensus.cc:383] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.849223 13997 raft_consensus.cc:738] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.849457 13997 consensus_queue.cc:260] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.849740 13999 tablet_bootstrap.cc:654] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.851956 13997 ts_tablet_manager.cc:1428] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250624 07:55:10.852895 14017 ts_tablet_manager.cc:1905] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:10.860293 13997 tablet_bootstrap.cc:492] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.861526 13998 tablet_bootstrap.cc:492] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.861788 13998 ts_tablet_manager.cc:1397] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.008s
I20250624 07:55:10.862630 13998 raft_consensus.cc:357] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.864334 13997 tablet_bootstrap.cc:654] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.864535 13998 raft_consensus.cc:383] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.864688 13998 raft_consensus.cc:738] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.865073 13998 consensus_queue.cc:260] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.865674 14015 ts_tablet_manager.cc:1918] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250624 07:55:10.865717 13998 ts_tablet_manager.cc:1428] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250624 07:55:10.865818 14015 log.cc:1199] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/325d48f2c20143129d28d0daacc0908a
I20250624 07:55:10.869500 14017 ts_tablet_manager.cc:1918] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250624 07:55:10.869625 13998 tablet_bootstrap.cc:492] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:10.869645 14017 log.cc:1199] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/325d48f2c20143129d28d0daacc0908a
I20250624 07:55:10.869980 14017 ts_tablet_manager.cc:1939] T 325d48f2c20143129d28d0daacc0908a P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:10.871382 13998 tablet_bootstrap.cc:654] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.871421 13997 tablet_bootstrap.cc:492] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.871582 13999 tablet_bootstrap.cc:492] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.871580 13997 ts_tablet_manager.cc:1397] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20250624 07:55:10.871778 13999 ts_tablet_manager.cc:1397] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20250624 07:55:10.872326 13997 raft_consensus.cc:357] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.872516 13997 raft_consensus.cc:383] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.872596 13997 raft_consensus.cc:738] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.872767 13999 raft_consensus.cc:357] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.872893 13997 consensus_queue.cc:260] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.872988 13999 raft_consensus.cc:383] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.873155 13999 raft_consensus.cc:738] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.873425 13999 consensus_queue.cc:260] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.874084 13999 ts_tablet_manager.cc:1428] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250624 07:55:10.874244 14015 ts_tablet_manager.cc:1939] T 325d48f2c20143129d28d0daacc0908a P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:10.878170 13265 catalog_manager.cc:4928] TS dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271): tablet 325d48f2c20143129d28d0daacc0908a (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]) successfully deleted
I20250624 07:55:10.878623 13997 ts_tablet_manager.cc:1428] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250624 07:55:10.880546 13997 tablet_bootstrap.cc:492] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.882404 13997 tablet_bootstrap.cc:654] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.883687 13264 catalog_manager.cc:4928] TS ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): tablet 325d48f2c20143129d28d0daacc0908a (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]) successfully deleted
W20250624 07:55:10.890353 13660 tablet.cc:2378] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 07:55:10.891234 13998 tablet_bootstrap.cc:492] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.891247 13997 tablet_bootstrap.cc:492] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.891393 13997 ts_tablet_manager.cc:1397] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250624 07:55:10.891403 13998 ts_tablet_manager.cc:1397] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.001s
I20250624 07:55:10.892172 13997 raft_consensus.cc:357] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.892361 13997 raft_consensus.cc:383] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.892513 13997 raft_consensus.cc:738] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.892526 13998 raft_consensus.cc:357] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.892706 13998 raft_consensus.cc:383] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.892799 13998 raft_consensus.cc:738] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.892925 13997 consensus_queue.cc:260] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.893049 13998 consensus_queue.cc:260] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.893558 13997 ts_tablet_manager.cc:1428] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250624 07:55:10.897869 13998 ts_tablet_manager.cc:1428] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250624 07:55:10.898224 13277 catalog_manager.cc:2149] created HMS catalog entry for table [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:10.900702 13998 tablet_bootstrap.cc:492] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:10.902418 13998 tablet_bootstrap.cc:654] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.911754 13998 tablet_bootstrap.cc:492] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.911952 13998 ts_tablet_manager.cc:1397] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20250624 07:55:10.912719 13998 raft_consensus.cc:357] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.913378 13998 raft_consensus.cc:383] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.913472 13998 raft_consensus.cc:738] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.913753 13998 consensus_queue.cc:260] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.917232 13590 tablet_service.cc:1468] Processing CreateTablet for tablet 6d7a465332db49218a4bb64803056fb3 (DEFAULT_TABLE table=default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.917661 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7a465332db49218a4bb64803056fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.921257 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 6d7a465332db49218a4bb64803056fb3 (DEFAULT_TABLE table=default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.921700 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7a465332db49218a4bb64803056fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.925307 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 6d7a465332db49218a4bb64803056fb3 (DEFAULT_TABLE table=default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:10.925750 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7a465332db49218a4bb64803056fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:10.929025 13277 catalog_manager.cc:5582] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab43ace4a8b44b908d247cad235e06df (127.12.180.2). New cstate: current_term: 1 leader_uuid: "ab43ace4a8b44b908d247cad235e06df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } health_report { overall_health: UNKNOWN } } }
I20250624 07:55:10.931882 13999 tablet_bootstrap.cc:492] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:10.932143 13997 tablet_bootstrap.cc:492] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:10.933463 13999 tablet_bootstrap.cc:654] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.933732 13997 tablet_bootstrap.cc:654] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.937783 13998 ts_tablet_manager.cc:1428] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20250624 07:55:10.941720 13998 tablet_bootstrap.cc:492] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:10.943495 13998 tablet_bootstrap.cc:654] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.946524 13997 tablet_bootstrap.cc:492] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:10.946821 13997 ts_tablet_manager.cc:1397] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20250624 07:55:10.948084 13997 raft_consensus.cc:357] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.948424 13997 raft_consensus.cc:383] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.948554 13997 raft_consensus.cc:738] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.949024 13997 consensus_queue.cc:260] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.949831 13997 ts_tablet_manager.cc:1428] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250624 07:55:10.963207 13998 tablet_bootstrap.cc:492] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.963467 13998 ts_tablet_manager.cc:1397] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20250624 07:55:10.964376 13998 raft_consensus.cc:357] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.964648 13998 raft_consensus.cc:383] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.964782 13998 raft_consensus.cc:738] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.965107 13998 consensus_queue.cc:260] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.968943 13277 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_50b9eb18e89f4d0a9d1b64950157ffe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:10.970029 13999 tablet_bootstrap.cc:492] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:10.970307 13999 ts_tablet_manager.cc:1397] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.000s
I20250624 07:55:10.971176 13999 raft_consensus.cc:357] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.971475 13999 raft_consensus.cc:383] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.971619 13999 raft_consensus.cc:738] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.971935 13998 ts_tablet_manager.cc:1428] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250624 07:55:10.971876 13999 consensus_queue.cc:260] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.972278 13998 tablet_bootstrap.cc:492] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:10.973301 13999 ts_tablet_manager.cc:1428] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250624 07:55:10.974045 13998 tablet_bootstrap.cc:654] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:10.985555 13273 catalog_manager.cc:2149] created HMS catalog entry for table [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:10.987747 13998 tablet_bootstrap.cc:492] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:10.987941 13998 ts_tablet_manager.cc:1397] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20250624 07:55:10.988724 13998 raft_consensus.cc:357] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.988950 13998 raft_consensus.cc:383] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:10.989022 13998 raft_consensus.cc:738] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:10.989248 13998 consensus_queue.cc:260] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:10.989780 13998 ts_tablet_manager.cc:1428] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250624 07:55:10.989996 14007 raft_consensus.cc:491] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:10.990193 14007 raft_consensus.cc:513] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:10.991888 14007 leader_election.cc:290] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.018972 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75bf5e310d42413ca5da83762bf91e9d" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df" is_pre_election: true
I20250624 07:55:11.019367 13749 raft_consensus.cc:2466] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 0.
I20250624 07:55:11.020211 13545 leader_election.cc:304] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:11.020645 14007 raft_consensus.cc:2802] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.020774 14007 raft_consensus.cc:491] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.020905 14007 raft_consensus.cc:3058] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.021999 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 5daee63b211a4b099533045210d213c8 (DEFAULT_TABLE table=default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.002128 13273 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_258e5344814340e49754e36a4631c5e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:11.022455 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5daee63b211a4b099533045210d213c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.023600 14007 raft_consensus.cc:513] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.024327 14007 leader_election.cc:290] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 election: Requested vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.025158 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75bf5e310d42413ca5da83762bf91e9d" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df"
I20250624 07:55:11.025345 13749 raft_consensus.cc:3058] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.026839 13590 tablet_service.cc:1468] Processing CreateTablet for tablet 5daee63b211a4b099533045210d213c8 (DEFAULT_TABLE table=default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.027204 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5daee63b211a4b099533045210d213c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.028074 13749 raft_consensus.cc:2466] T 75bf5e310d42413ca5da83762bf91e9d P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 1.
I20250624 07:55:11.028591 13545 leader_election.cc:304] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:11.029170 14007 raft_consensus.cc:2802] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:11.031935 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 5daee63b211a4b099533045210d213c8 (DEFAULT_TABLE table=default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.032384 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5daee63b211a4b099533045210d213c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.040611 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75bf5e310d42413ca5da83762bf91e9d" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:11.040917 13885 raft_consensus.cc:2466] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 0.
I20250624 07:55:11.041286 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75bf5e310d42413ca5da83762bf91e9d" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:11.041556 13884 raft_consensus.cc:3058] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.045775 13884 raft_consensus.cc:2466] T 75bf5e310d42413ca5da83762bf91e9d P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 1.
I20250624 07:55:11.054612 13997 tablet_bootstrap.cc:492] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:11.056429 13997 tablet_bootstrap.cc:654] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:11.058053 13998 tablet_bootstrap.cc:492] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:11.059373 13999 tablet_bootstrap.cc:492] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:11.059795 13998 tablet_bootstrap.cc:654] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:11.061470 13999 tablet_bootstrap.cc:654] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:11.067412 14007 raft_consensus.cc:695] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [term 1 LEADER]: Becoming Leader. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Running, Role: LEADER
I20250624 07:55:11.067896 14007 consensus_queue.cc:237] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.070861 13999 tablet_bootstrap.cc:492] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:11.071015 13999 ts_tablet_manager.cc:1397] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20250624 07:55:11.071774 13999 raft_consensus.cc:357] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.071987 13999 raft_consensus.cc:383] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.072094 13999 raft_consensus.cc:738] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.072362 13999 consensus_queue.cc:260] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.072572 13998 tablet_bootstrap.cc:492] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:11.072746 13998 ts_tablet_manager.cc:1397] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20250624 07:55:11.072960 13999 ts_tablet_manager.cc:1428] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250624 07:55:11.073508 13998 raft_consensus.cc:357] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.073712 13998 raft_consensus.cc:383] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.073820 13998 raft_consensus.cc:738] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.074041 13998 consensus_queue.cc:260] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.074569 13998 ts_tablet_manager.cc:1428] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250624 07:55:11.078465 13997 tablet_bootstrap.cc:492] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:11.078660 13997 ts_tablet_manager.cc:1397] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.013s
I20250624 07:55:11.079429 13997 raft_consensus.cc:357] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.079672 13997 raft_consensus.cc:383] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.079805 13997 raft_consensus.cc:738] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.080062 13997 consensus_queue.cc:260] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.080590 13997 ts_tablet_manager.cc:1428] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250624 07:55:11.088364 14003 raft_consensus.cc:491] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.088577 14003 raft_consensus.cc:513] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.089221 14003 leader_election.cc:290] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.090215 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82f70c673a9749be9fc70112188be360" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.090454 13612 raft_consensus.cc:2466] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.090814 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82f70c673a9749be9fc70112188be360" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:11.091048 13884 raft_consensus.cc:2466] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.091568 13683 leader_election.cc:304] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:11.092321 14003 raft_consensus.cc:2802] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.092423 14003 raft_consensus.cc:491] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.092514 14003 raft_consensus.cc:3058] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.094607 14025 raft_consensus.cc:491] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.094741 14025 raft_consensus.cc:513] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.095537 14025 leader_election.cc:290] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.096387 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b765ab702f4817879ab30ebb558336" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.096586 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b765ab702f4817879ab30ebb558336" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:11.096669 13612 raft_consensus.cc:2466] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.096781 13884 raft_consensus.cc:2466] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.097409 13683 leader_election.cc:304] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:11.097929 14025 raft_consensus.cc:2802] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.098028 14025 raft_consensus.cc:491] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.098098 14025 raft_consensus.cc:3058] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.100836 14025 raft_consensus.cc:513] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.101948 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b765ab702f4817879ab30ebb558336" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:11.102146 13612 raft_consensus.cc:3058] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.104774 13612 raft_consensus.cc:2466] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.105386 13683 leader_election.cc:304] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:11.105696 14023 raft_consensus.cc:2802] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:11.106889 14003 raft_consensus.cc:513] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.107103 14023 raft_consensus.cc:695] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Becoming Leader. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Running, Role: LEADER
I20250624 07:55:11.108508 14023 consensus_queue.cc:237] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.110409 14025 leader_election.cc:290] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Requested vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.111378 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82f70c673a9749be9fc70112188be360" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:11.111600 13885 raft_consensus.cc:3058] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.114621 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b765ab702f4817879ab30ebb558336" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:11.114872 13884 raft_consensus.cc:3058] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.119068 13885 raft_consensus.cc:2466] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.119427 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82f70c673a9749be9fc70112188be360" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:11.119699 13612 raft_consensus.cc:3058] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.125506 13683 leader_election.cc:304] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.125813 14023 raft_consensus.cc:2802] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:11.125960 14023 raft_consensus.cc:695] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Becoming Leader. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Running, Role: LEADER
I20250624 07:55:11.127193 14023 consensus_queue.cc:237] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.125033 13884 raft_consensus.cc:2466] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.110432 14003 leader_election.cc:290] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Requested vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.130553 13612 raft_consensus.cc:2466] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
07:55:11.195 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_e87f394359f24830b4171bf6b60d4ab2 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20250624 07:55:11.197749 13275 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=76494ba8f4fe4b08ab272bd6d0cc8abf]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_e87f394359f24830b4171bf6b60d4ab2 already exists)
I20250624 07:55:11.201584 13275 catalog_manager.cc:5582] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbb2615829f748ef929761ed90057cb7 (127.12.180.1). New cstate: current_term: 1 leader_uuid: "dbb2615829f748ef929761ed90057cb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } health_report { overall_health: UNKNOWN } } }
I20250624 07:55:11.211584 13275 catalog_manager.cc:5582] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab43ace4a8b44b908d247cad235e06df (127.12.180.2). New cstate: current_term: 1 leader_uuid: "ab43ace4a8b44b908d247cad235e06df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } health_report { overall_health: HEALTHY } } }
I20250624 07:55:11.212250 13275 catalog_manager.cc:5582] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab43ace4a8b44b908d247cad235e06df (127.12.180.2). New cstate: current_term: 1 leader_uuid: "ab43ace4a8b44b908d247cad235e06df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } health_report { overall_health: UNKNOWN } } }
I20250624 07:55:11.217329 14023 raft_consensus.cc:491] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.217511 14023 raft_consensus.cc:513] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.219075 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31617090f65045f08d94b793ff525275" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.219326 13612 raft_consensus.cc:2466] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.219947 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31617090f65045f08d94b793ff525275" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:11.220148 13884 raft_consensus.cc:2466] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.220924 13683 leader_election.cc:304] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.221302 14003 raft_consensus.cc:2802] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.221408 14003 raft_consensus.cc:491] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.221490 14003 raft_consensus.cc:3058] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.224274 14003 raft_consensus.cc:513] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.224908 14003 leader_election.cc:290] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Requested vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.229929 13275 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48636:
table { table_name: "default.Table_8a84bf94dbd94e65946d5e56b69172be" } new_table_name: "default.Table_d133c18e2676484dabe455c76b424d05" modify_external_catalogs: true
I20250624 07:55:11.232616 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31617090f65045f08d94b793ff525275" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:11.232852 13884 raft_consensus.cc:3058] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.235659 13884 raft_consensus.cc:2466] T 31617090f65045f08d94b793ff525275 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.235862 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31617090f65045f08d94b793ff525275" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:11.236121 13612 raft_consensus.cc:3058] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.236452 13683 leader_election.cc:304] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.236742 14025 raft_consensus.cc:491] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.236865 14025 raft_consensus.cc:513] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.237437 14025 leader_election.cc:290] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f96059407ad14a70971e622891d2f99c (127.12.180.3:33179), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:11.237564 14025 raft_consensus.cc:2802] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:11.237682 14025 raft_consensus.cc:695] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Becoming Leader. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Running, Role: LEADER
I20250624 07:55:11.237929 14025 consensus_queue.cc:237] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.238301 14023 leader_election.cc:290] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.239068 13612 raft_consensus.cc:2466] T 31617090f65045f08d94b793ff525275 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.241582 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d7a465332db49218a4bb64803056fb3" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:11.241803 13885 raft_consensus.cc:2466] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.243400 13683 leader_election.cc:304] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.243695 14025 raft_consensus.cc:2802] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.243782 14025 raft_consensus.cc:491] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.243814 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d7a465332db49218a4bb64803056fb3" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.243891 14025 raft_consensus.cc:3058] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.243999 13612 raft_consensus.cc:2466] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.245450 13271 catalog_manager.cc:2149] created HMS catalog entry for table [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:11.251461 14005 raft_consensus.cc:491] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.251670 14005 raft_consensus.cc:513] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.253329 14005 leader_election.cc:290] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:11.256914 14025 raft_consensus.cc:513] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.258018 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d7a465332db49218a4bb64803056fb3" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:11.258349 13884 raft_consensus.cc:3058] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.261233 13884 raft_consensus.cc:2466] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.261802 13683 leader_election.cc:304] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.262187 14003 raft_consensus.cc:2802] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:11.262351 14003 raft_consensus.cc:695] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Becoming Leader. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Running, Role: LEADER
I20250624 07:55:11.262629 14003 consensus_queue.cc:237] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.276609 14025 leader_election.cc:290] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Requested vote from peers f96059407ad14a70971e622891d2f99c (127.12.180.3:33179), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:11.280853 13280 catalog_manager.cc:2149] created HMS catalog entry for table [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:11.282984 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d7a465332db49218a4bb64803056fb3" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:11.283214 13612 raft_consensus.cc:3058] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.285904 13271 catalog_manager.cc:5582] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab43ace4a8b44b908d247cad235e06df (127.12.180.2). New cstate: current_term: 1 leader_uuid: "ab43ace4a8b44b908d247cad235e06df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } health_report { overall_health: UNKNOWN } } }
I20250624 07:55:11.288122 13612 raft_consensus.cc:2466] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
F20250624 07:55:11.290200 13499 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250624 07:55:11.292425 13280 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48636 (request call id 13) took 1012 ms. Trace:
@ 0x7f9191cd2dcd google::LogMessage::Fail() at ??:0
I20250624 07:55:11.294553 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c95dc3a9620745c6a7020ca7434e2c1e" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.294883 13612 raft_consensus.cc:2391] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f96059407ad14a70971e622891d2f99c in current term 1: Already voted for candidate ab43ace4a8b44b908d247cad235e06df in this term.
I20250624 07:55:11.297295 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c95dc3a9620745c6a7020ca7434e2c1e" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df" is_pre_election: true
I20250624 07:55:11.298255 13818 leader_election.cc:304] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f96059407ad14a70971e622891d2f99c; no voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7
I20250624 07:55:11.299027 14005 raft_consensus.cc:3058] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.301849 14005 raft_consensus.cc:2747] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250624 07:55:11.305281 13271 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48636:
name: "default.Table_ec6cab3afe9e48f3a38fa9c0c2ca3922"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:11.311681 13590 tablet_service.cc:1468] Processing CreateTablet for tablet 2b11e2bf606e4e94a579bcf49502d94b (DEFAULT_TABLE table=default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.312140 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b11e2bf606e4e94a579bcf49502d94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.316965 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 2b11e2bf606e4e94a579bcf49502d94b (DEFAULT_TABLE table=default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.317375 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b11e2bf606e4e94a579bcf49502d94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.318709 13729 tablet_service.cc:1468] Processing CreateTablet for tablet d377dcea0fc1444abed05b5f8fa6b32c (DEFAULT_TABLE table=default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.318897 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 2b11e2bf606e4e94a579bcf49502d94b (DEFAULT_TABLE table=default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.319063 13729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d377dcea0fc1444abed05b5f8fa6b32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.319094 13591 tablet_service.cc:1468] Processing CreateTablet for tablet d377dcea0fc1444abed05b5f8fa6b32c (DEFAULT_TABLE table=default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.319260 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b11e2bf606e4e94a579bcf49502d94b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.319416 13591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d377dcea0fc1444abed05b5f8fa6b32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.323053 13865 tablet_service.cc:1468] Processing CreateTablet for tablet d377dcea0fc1444abed05b5f8fa6b32c (DEFAULT_TABLE table=default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:11.323411 13865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d377dcea0fc1444abed05b5f8fa6b32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:11.292624 13280 rpcz_store.cc:276] 0624 07:55:10.282833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0624 07:55:10.282883 (+ 50us) service_pool.cc:225] Handling call
0624 07:55:10.459090 (+176207us) catalog_manager.cc:2076] Acquired catalog manager lock
0624 07:55:10.459519 (+ 429us) catalog_manager.cc:2122] Created new table and tablet info
0624 07:55:11.280770 (+821251us) catalog_manager.cc:2148] Created new table in HMS catalog
0624 07:55:11.292101 (+ 11331us) catalog_manager.cc:2182] Wrote table and tablets to system table
0624 07:55:11.292248 (+ 147us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0624 07:55:11.292375 (+ 127us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0624 07:55:11.281917 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0624 07:55:11.282041 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0624 07:55:11.282049 (+ 8us) write_op.cc:435] Acquired schema lock
0624 07:55:11.282051 (+ 2us) tablet.cc:660] Decoding operations
0624 07:55:11.282147 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0624 07:55:11.282161 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0624 07:55:11.282188 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
0624 07:55:11.282273 (+ 85us) tablet.cc:699] Row locks acquired
0624 07:55:11.282276 (+ 3us) write_op.cc:260] PREPARE: finished
0624 07:55:11.282320 (+ 44us) write_op.cc:270] Start()
0624 07:55:11.282366 (+ 46us) write_op.cc:276] Timestamp: P: 1750751711282308 usec, L: 0
0624 07:55:11.282370 (+ 4us) op_driver.cc:348] REPLICATION: starting
0624 07:55:11.286866 (+ 4496us) log.cc:844] Serialized 432 byte log entry
0624 07:55:11.287235 (+ 369us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x616000040f54. stack: 00007f9194a07336 00007f9194a070f0 00007f91939fd3ad 0000000000971004 0000000000970fcd 000000000097099a 00007f91a043c762 00007f91a17e3855 00007f91a17e2379 00007f91a17ea7a0 00007f91a17ea5ad 0000000000a7573c 00007f9194a4c9df 00007f9194a4fcab 00007f9194a4fb28 0000000000a7573c
0624 07:55:11.290652 (+ 3417us) op_driver.cc:464] REPLICATION: finished
0624 07:55:11.291374 (+ 722us) write_op.cc:301] APPLY: starting
0624 07:55:11.291396 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0624 07:55:11.291449 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0624 07:55:11.291454 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0624 07:55:11.291630 (+ 176us) tablet.cc:1381] finished ApplyRowOperation cycle
0624 07:55:11.291676 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0624 07:55:11.291687 (+ 11us) write_op.cc:312] APPLY: finished
0624 07:55:11.291783 (+ 96us) log.cc:844] Serialized 28 byte log entry
0624 07:55:11.291892 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0624 07:55:11.291924 (+ 32us) write_op.cc:454] Released schema lock
0624 07:55:11.291932 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0624 07:55:11.298461 (+ 6529us) write_op.cc:489] Releasing partition, row and schema locks
0624 07:55:11.298468 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":785735,"Hive Metastore.run_cpu_time_us":285,"Hive Metastore.run_wall_time_us":35370,"spinlock_wait_cycles":4608,"child_traces":[["op",{"apply.queue_time_us":655,"apply.run_cpu_time_us":649,"apply.run_wall_time_us":649,"num_ops":2,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":7391,"prepare.run_wall_time_us":16538,"raft.queue_time_us":1420,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":8245,"spinlock_wait_cycles":9793920,"thread_start_us":10660,"threads_started":1}]]}
I20250624 07:55:11.324568 13280 catalog_manager.cc:5582] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab43ace4a8b44b908d247cad235e06df (127.12.180.2). New cstate: current_term: 1 leader_uuid: "ab43ace4a8b44b908d247cad235e06df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } health_report { overall_health: HEALTHY } } }
I20250624 07:55:11.325683 13998 tablet_bootstrap.cc:492] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
@ 0x7f9191cd6b93 google::LogMessage::SendToLog() at ??:0
I20250624 07:55:11.327277 13998 tablet_bootstrap.cc:654] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
@ 0x7f9191cd27cc google::LogMessage::Flush() at ??:0
@ 0x7f9191cd2a11 google::LogMessage::~LogMessage() at ??:0
I20250624 07:55:11.338459 13999 tablet_bootstrap.cc:492] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:11.340639 13999 tablet_bootstrap.cc:654] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
@ 0x7f91a80d36d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250624 07:55:11.346814 13997 tablet_bootstrap.cc:492] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:11.348003 13280 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48636:
table { table_name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe" } modify_external_catalogs: true
I20250624 07:55:11.348249 13280 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48636:
table { table_name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe" } modify_external_catalogs: true
I20250624 07:55:11.348526 13997 tablet_bootstrap.cc:654] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
@ 0x7f91a80d1f81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f91a80d9bfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f91a80d9a79 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa7573d std::function<>::operator()() at ??:?
@ 0x7f9194a2c4fe kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9197e4a6db start_thread at ??:0
@ 0x7f9190b8371f clone at ??:0
I20250624 07:55:11.381508 14025 raft_consensus.cc:491] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.381711 14025 raft_consensus.cc:513] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.382496 14025 leader_election.cc:290] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f96059407ad14a70971e622891d2f99c (127.12.180.3:33179), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:11.383232 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5daee63b211a4b099533045210d213c8" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:11.383452 13884 raft_consensus.cc:2466] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.383508 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5daee63b211a4b099533045210d213c8" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.383782 13612 raft_consensus.cc:2466] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:11.384338 13683 leader_election.cc:304] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.384670 14025 raft_consensus.cc:2802] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.384770 14025 raft_consensus.cc:491] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.384872 14025 raft_consensus.cc:3058] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.385723 13999 tablet_bootstrap.cc:492] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:11.385962 13999 ts_tablet_manager.cc:1397] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s
I20250624 07:55:11.387072 13999 raft_consensus.cc:357] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.387313 13999 raft_consensus.cc:383] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.387401 13999 raft_consensus.cc:738] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.387789 14025 raft_consensus.cc:513] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.388425 14025 leader_election.cc:290] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Requested vote from peers f96059407ad14a70971e622891d2f99c (127.12.180.3:33179), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:11.389254 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5daee63b211a4b099533045210d213c8" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:11.389495 13612 raft_consensus.cc:3058] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.390506 13884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5daee63b211a4b099533045210d213c8" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:11.390733 13884 raft_consensus.cc:3058] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.392879 13612 raft_consensus.cc:2466] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.394043 13683 leader_election.cc:304] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:11.394657 13884 raft_consensus.cc:2466] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:11.394850 14025 raft_consensus.cc:2802] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:11.395524 14025 raft_consensus.cc:695] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Becoming Leader. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Running, Role: LEADER
I20250624 07:55:11.395857 14025 consensus_queue.cc:237] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:11.401060 13997 tablet_bootstrap.cc:492] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:11.401266 13997 ts_tablet_manager.cc:1397] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.007s
I20250624 07:55:11.402487 13997 raft_consensus.cc:357] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.402721 13997 raft_consensus.cc:383] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.402796 13997 raft_consensus.cc:738] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.407509 13997 consensus_queue.cc:260] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.408515 13997 ts_tablet_manager.cc:1428] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250624 07:55:11.408919 13997 tablet_bootstrap.cc:492] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:11.409592 13999 consensus_queue.cc:260] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.413451 13999 ts_tablet_manager.cc:1428] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.027s user 0.000s sys 0.003s
I20250624 07:55:11.419308 13998 tablet_bootstrap.cc:492] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:11.419478 13998 ts_tablet_manager.cc:1397] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.094s user 0.027s sys 0.054s
I20250624 07:55:11.420220 13998 raft_consensus.cc:357] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.420418 13998 raft_consensus.cc:383] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.420485 13998 raft_consensus.cc:738] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.420625 13999 tablet_bootstrap.cc:492] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:11.420753 13998 consensus_queue.cc:260] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
W20250624 07:55:11.420933 14014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:11.421339 14014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:11.421460 14014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250624 07:55:11.422787 13997 tablet_bootstrap.cc:654] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:11.423213 13998 ts_tablet_manager.cc:1428] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250624 07:55:11.423555 13998 tablet_bootstrap.cc:492] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:11.425081 13999 tablet_bootstrap.cc:654] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:11.426476 13998 tablet_bootstrap.cc:654] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:11.434301 13998 tablet_bootstrap.cc:492] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:11.434492 13998 ts_tablet_manager.cc:1397] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
W20250624 07:55:11.435189 14014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:11.435292 14014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:11.435369 14014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:11.435410 14014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:11.435477 13998 raft_consensus.cc:357] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.435691 13998 raft_consensus.cc:383] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.435784 13998 raft_consensus.cc:738] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.436044 13998 consensus_queue.cc:260] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.438688 13998 ts_tablet_manager.cc:1428] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250624 07:55:11.440295 13999 tablet_bootstrap.cc:492] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:11.440433 13999 ts_tablet_manager.cc:1397] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20250624 07:55:11.441188 13999 raft_consensus.cc:357] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.441375 13999 raft_consensus.cc:383] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.441449 13999 raft_consensus.cc:738] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.441675 13999 consensus_queue.cc:260] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.443439 13999 ts_tablet_manager.cc:1428] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250624 07:55:11.445794 13997 tablet_bootstrap.cc:492] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:11.445966 13997 ts_tablet_manager.cc:1397] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s
I20250624 07:55:11.446770 13997 raft_consensus.cc:357] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.446900 13997 raft_consensus.cc:383] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:11.446959 13997 raft_consensus.cc:738] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.447137 13997 consensus_queue.cc:260] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.447629 13997 ts_tablet_manager.cc:1428] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 07:55:11.451814 14014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.180.60
--webserver_port=42595
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:11.453967 14014 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:11.456452 14014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:11.478449 14070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:11.482232 14068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:11.482475 14074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:11.486667 14014 server_base.cc:1048] running on GCE node
I20250624 07:55:11.487478 14014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:11.488957 14014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:11.490159 14014 hybrid_clock.cc:648] HybridClock initialized: now 1750751711490114 us; error 48 us; skew 500 ppm
I20250624 07:55:11.490522 14014 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:11.493041 14014 webserver.cc:469] Webserver started at http://127.12.180.60:42595/ using document root <none> and password file <none>
I20250624 07:55:11.493865 14014 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:11.493959 14014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:11.500164 14014 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20250624 07:55:11.503911 14081 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:11.505684 14014 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250624 07:55:11.505874 14014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
uuid: "632ba412a681447e93af32d7fdf0691d"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:11.506800 14014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:11.544845 14014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:11.545951 14014 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:11.546308 14014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:11.571183 14014 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.60:33449
I20250624 07:55:11.571202 14135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.60:33449 every 8 connection(s)
I20250624 07:55:11.573208 14014 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
I20250624 07:55:11.577711 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 14014
I20250624 07:55:11.582100 14136 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:11.588133 14136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap starting.
I20250624 07:55:11.596437 14005 raft_consensus.cc:491] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.596647 14005 raft_consensus.cc:513] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.597332 14005 leader_election.cc:290] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:11.598219 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b11e2bf606e4e94a579bcf49502d94b" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.598310 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b11e2bf606e4e94a579bcf49502d94b" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df" is_pre_election: true
I20250624 07:55:11.598448 13612 raft_consensus.cc:2466] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:11.598489 13749 raft_consensus.cc:2466] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:11.598925 13819 leader_election.cc:304] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb2615829f748ef929761ed90057cb7, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.599637 14005 raft_consensus.cc:2802] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.599727 14005 raft_consensus.cc:491] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.599841 14005 raft_consensus.cc:3058] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.601620 14005 raft_consensus.cc:513] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.601934 14136 log.cc:826] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:11.602680 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b11e2bf606e4e94a579bcf49502d94b" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:11.602914 13612 raft_consensus.cc:3058] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.603701 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b11e2bf606e4e94a579bcf49502d94b" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df"
I20250624 07:55:11.603902 13749 raft_consensus.cc:3058] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.605350 13612 raft_consensus.cc:2466] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
I20250624 07:55:11.605821 13819 leader_election.cc:304] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb2615829f748ef929761ed90057cb7, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.605890 14005 leader_election.cc:290] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Requested vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:11.606220 14005 raft_consensus.cc:2802] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:11.606797 13749 raft_consensus.cc:2466] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
I20250624 07:55:11.608812 14005 raft_consensus.cc:695] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 1 LEADER]: Becoming Leader. State: Replica: f96059407ad14a70971e622891d2f99c, State: Running, Role: LEADER
I20250624 07:55:11.609947 14005 consensus_queue.cc:237] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.615924 14136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:55:11.616549 14136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap complete.
I20250624 07:55:11.626220 14136 raft_consensus.cc:357] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:11.627161 14136 raft_consensus.cc:738] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Initialized, Role: FOLLOWER
I20250624 07:55:11.627929 14136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:11.628609 13930 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:11.628862 13930 heartbeater.cc:499] Master 127.12.180.60:33449 was elected leader, sending a full tablet report...
I20250624 07:55:11.629076 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:11.629354 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:11.630434 14136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:11.644603 14157 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:11.644711 14157 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: loading cluster ID for follower catalog manager: success
I20250624 07:55:11.649025 14157 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: acquiring CA information for follower catalog manager: success
I20250624 07:55:11.651542 14157 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250624 07:55:11.670572 13940 connection.cc:537] client connection to 127.12.180.62:44637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:11.671114 13821 connection.cc:537] client connection to 127.12.180.62:44637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:11.671376 13940 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.180.62:44637)
W20250624 07:55:11.671813 13940 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.180.62:44637)
W20250624 07:55:11.672384 13685 connection.cc:537] client connection to 127.12.180.62:44637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:11.672447 13940 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.180.62:44637)
W20250624 07:55:11.673417 13940 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.180.62:44637)
W20250624 07:55:11.673955 14048 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:11.672049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.180.62:44637 (local address 127.12.180.3:52253)
0624 07:55:11.672260 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0624 07:55:11.672282 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0624 07:55:11.672413 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0624 07:55:11.673043 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":77}
W20250624 07:55:11.675864 14159 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:11.639140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.180.62:44637 (local address 127.0.0.1:48642)
0624 07:55:11.639764 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0624 07:55:11.639809 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0624 07:55:11.639991 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0624 07:55:11.673439 (+ 33448us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":9938,"threads_started":1}
W20250624 07:55:11.676700 13548 connection.cc:537] client connection to 127.12.180.62:44637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 07:55:11.676743 14014 master_runner.cc:186] Error getting master registration for 127.12.180.62:44637: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250624 07:55:11.682132 14014 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:11.713943 14025 consensus_queue.cc:1035] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:11.728881 13657 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:11.732988 14101 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:50845; Asking this server to re-register.
I20250624 07:55:11.734019 13657 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:11.734266 14003 consensus_queue.cc:1035] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:11.736387 14100 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:11.736614 13930 heartbeater.cc:499] Master 127.12.180.60:33449 was elected leader, sending a full tablet report...
I20250624 07:55:11.739920 13794 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:11.740114 13794 heartbeater.cc:499] Master 127.12.180.60:33449 was elected leader, sending a full tablet report...
I20250624 07:55:11.742901 14100 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:37135; Asking this server to re-register.
I20250624 07:55:11.743881 13794 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:11.745831 14100 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:11.748739 14100 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:49805; Asking this server to re-register.
I20250624 07:55:11.749610 13930 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:11.751220 14100 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:11.830617 14007 consensus_queue.cc:1035] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:11.844951 14171 consensus_queue.cc:1035] T 75bf5e310d42413ca5da83762bf91e9d P dbb2615829f748ef929761ed90057cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:11.854890 14173 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 588e956d4ac040deb3c06e57bd3c6b88)
I20250624 07:55:11.855255 14173 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:11.871510 14173 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:11.885777 13337 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
I20250624 07:55:11.885680 14111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "632ba412a681447e93af32d7fdf0691d" is_pre_election: true
I20250624 07:55:11.886395 14111 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52afd679ffb241a3a9022cc850c03493 in term 1.
I20250624 07:55:11.887041 13334 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d; no voters: 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:11.887356 14173 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250624 07:55:11.887476 14173 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 588e956d4ac040deb3c06e57bd3c6b88)
I20250624 07:55:11.887579 14173 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:55:11.889555 14173 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:11.890192 14173 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 3 election: Requested vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:11.890772 14111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "632ba412a681447e93af32d7fdf0691d"
I20250624 07:55:11.890962 14111 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 1 FOLLOWER]: Advancing to term 3
W20250624 07:55:11.895777 13337 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
I20250624 07:55:11.896605 14111 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52afd679ffb241a3a9022cc850c03493 in term 3.
I20250624 07:55:11.897019 13334 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d; no voters: 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:11.897384 14173 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 FOLLOWER]: Leader election won for term 3
I20250624 07:55:11.898393 14173 raft_consensus.cc:695] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 LEADER]: Becoming Leader. State: Replica: 52afd679ffb241a3a9022cc850c03493, State: Running, Role: LEADER
I20250624 07:55:11.899031 14173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:11.901448 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52afd679ffb241a3a9022cc850c03493. Latest consensus state: current_term: 3 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:11.901789 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:11.902762 14178 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:11.905872 14178 catalog_manager.cc:671] Loaded metadata for table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:11.906491 14178 catalog_manager.cc:671] Loaded metadata for table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:11.906906 14178 catalog_manager.cc:671] Loaded metadata for table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:11.907265 14178 catalog_manager.cc:671] Loaded metadata for table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:11.907656 14178 catalog_manager.cc:671] Loaded metadata for table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:11.908018 14178 catalog_manager.cc:671] Loaded metadata for table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:11.908358 14178 catalog_manager.cc:671] Loaded metadata for table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:11.908759 14178 catalog_manager.cc:671] Loaded metadata for table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:11.909098 14178 catalog_manager.cc:671] Loaded metadata for table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:11.911335 14178 tablet_loader.cc:96] loaded metadata for tablet 09b765ab702f4817879ab30ebb558336 (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72])
I20250624 07:55:11.911757 14178 tablet_loader.cc:96] loaded metadata for tablet 2b11e2bf606e4e94a579bcf49502d94b (table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d])
I20250624 07:55:11.912151 14178 tablet_loader.cc:96] loaded metadata for tablet 31617090f65045f08d94b793ff525275 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2])
I20250624 07:55:11.912675 14178 tablet_loader.cc:96] loaded metadata for tablet 5daee63b211a4b099533045210d213c8 (table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62])
I20250624 07:55:11.913031 14178 tablet_loader.cc:96] loaded metadata for tablet 6d7a465332db49218a4bb64803056fb3 (table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939])
I20250624 07:55:11.913384 14178 tablet_loader.cc:96] loaded metadata for tablet 75bf5e310d42413ca5da83762bf91e9d (table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b])
I20250624 07:55:11.913693 14178 tablet_loader.cc:96] loaded metadata for tablet 82f70c673a9749be9fc70112188be360 (table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e])
I20250624 07:55:11.914250 14178 tablet_loader.cc:96] loaded metadata for tablet c95dc3a9620745c6a7020ca7434e2c1e (table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e])
I20250624 07:55:11.914597 14178 tablet_loader.cc:96] loaded metadata for tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542])
I20250624 07:55:11.914867 14178 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:11.916149 14178 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:11.916242 14178 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:11.917822 14178 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:11.918890 14178 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Loaded TSK: 0
I20250624 07:55:11.919878 14178 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:11.921083 14178 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:11.922003 14178 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250624 07:55:11.939564 14003 consensus_queue.cc:1035] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:11.944751 14043 consensus_queue.cc:1035] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:11.947993 13659 heartbeater.cc:499] Master 127.12.180.61:46565 was elected leader, sending a full tablet report...
I20250624 07:55:11.955363 13932 heartbeater.cc:499] Master 127.12.180.61:46565 was elected leader, sending a full tablet report...
I20250624 07:55:11.958897 13349 catalog_manager.cc:5582] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c reported cstate change: term changed from 0 to 1, leader changed from <none> to f96059407ad14a70971e622891d2f99c (127.12.180.3). New cstate: current_term: 1 leader_uuid: "f96059407ad14a70971e622891d2f99c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } health_report { overall_health: HEALTHY } } }
I20250624 07:55:11.961282 13341 master_service.cc:948] ReplaceTablet: received request to replace tablet 75bf5e310d42413ca5da83762bf91e9d from {username='slave'} at 127.0.0.1:55102
I20250624 07:55:11.960384 13346 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55102:
table { table_name: "default.Table_953a2fe7f1694bb7b1afcbbaddd632b6" } modify_external_catalogs: true
I20250624 07:55:11.960914 13345 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55102:
name: "default.Table_6c5708ec0a8a44f4bc2b8d669d041a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:11.963482 13346 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55102:
table { table_name: "default.Table_953a2fe7f1694bb7b1afcbbaddd632b6" } modify_external_catalogs: true
I20250624 07:55:11.960757 13343 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55102:
name: "default.Table_fe124062a0864c5e9472552b1a471845"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:11.961059 13342 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55102:
table { table_name: "default.Table_6e36abec7aef4670b05e620d91bba579" } modify_external_catalogs: true
I20250624 07:55:11.963923 13342 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55102:
table { table_name: "default.Table_6e36abec7aef4670b05e620d91bba579" } modify_external_catalogs: true
I20250624 07:55:11.957739 13347 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55102:
table { table_name: "default.Table_02926d7fe80d4545a80eb082aa7d2671" } modify_external_catalogs: true
I20250624 07:55:11.961122 13344 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55102:
name: "default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:11.964527 13347 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55102:
table { table_name: "default.Table_02926d7fe80d4545a80eb082aa7d2671" } modify_external_catalogs: true
I20250624 07:55:11.961311 13348 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55102:
name: "default.Table_17819dd7286645d8a22ec7eed153297a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:11.960878 13350 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55102:
name: "default.Table_3e714bb6900c450b90b4519baaaa3aba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:11.969188 14111 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Refusing update from remote peer 52afd679ffb241a3a9022cc850c03493: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250624 07:55:11.970032 14177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 4, Time since last communication: 0.000s
I20250624 07:55:11.970531 14005 raft_consensus.cc:491] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:11.970681 14005 raft_consensus.cc:513] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.971333 14005 leader_election.cc:290] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:11.971724 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d377dcea0fc1444abed05b5f8fa6b32c" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:11.971957 13612 raft_consensus.cc:2466] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:11.971956 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d377dcea0fc1444abed05b5f8fa6b32c" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df" is_pre_election: true
I20250624 07:55:11.972152 13749 raft_consensus.cc:2466] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:11.972422 13819 leader_election.cc:304] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb2615829f748ef929761ed90057cb7, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.972779 14005 raft_consensus.cc:2802] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:11.972873 14005 raft_consensus.cc:491] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:11.972942 14005 raft_consensus.cc:3058] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.975687 14005 raft_consensus.cc:513] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:11.976682 13612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d377dcea0fc1444abed05b5f8fa6b32c" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:11.976895 13612 raft_consensus.cc:3058] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
W20250624 07:55:11.978613 13337 consensus_peers.cc:487] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 -> Peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Couldn't send request to peer 588e956d4ac040deb3c06e57bd3c6b88. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:55:11.978613 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d377dcea0fc1444abed05b5f8fa6b32c" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df"
I20250624 07:55:11.978828 13749 raft_consensus.cc:3058] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:11.979851 13612 raft_consensus.cc:2466] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
F20250624 07:55:11.980435 13527 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20250624 07:55:11.981606 13749 raft_consensus.cc:2466] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
I20250624 07:55:11.982153 13818 leader_election.cc:304] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:11.982586 14140 raft_consensus.cc:2802] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election won for term 1
@ 0x7f4972ddbdcd google::LogMessage::Fail() at ??:0
I20250624 07:55:11.982609 14005 leader_election.cc:290] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Requested vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
@ 0x7f4972ddfb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4972ddb7cc google::LogMessage::Flush() at ??:0
@ 0x7f4972ddba11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f49891dc6d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250624 07:55:11.997121 14140 raft_consensus.cc:695] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 1 LEADER]: Becoming Leader. State: Replica: f96059407ad14a70971e622891d2f99c, State: Running, Role: LEADER
I20250624 07:55:11.997540 14140 consensus_queue.cc:237] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
@ 0x7f49891daf81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250624 07:55:12.015116 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52afd679ffb241a3a9022cc850c03493. Latest consensus state: current_term: 3 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:12.015430 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:12.015583 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:12.015822 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:12.018936 14181 mvcc.cc:204] Tried to move back new op lower bound from 7171079012217434112 to 7171079011943358464. Current Snapshot: MvccSnapshot[applied={T|T < 7171079012217434112}]
I20250624 07:55:12.019743 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:12.019984 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:12.025370 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:12.025632 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: LEADER
@ 0x7f49891e2bfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250624 07:55:12.028450 13349 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55102:
name: "default.Table_c316e293c95c47eda371175ca862fe0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f49891e2a79 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa7573d std::function<>::operator()() at ??:?
I20250624 07:55:12.036787 13341 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Sending DeleteTablet for 3 replicas of tablet 75bf5e310d42413ca5da83762bf91e9d
@ 0x7f4975b354fe kudu::Thread::SuperviseThread() at ??:0
I20250624 07:55:12.038915 13341 catalog_manager.cc:6652] ReplaceTablet: tablet 75bf5e310d42413ca5da83762bf91e9d deleted and replaced by tablet 8ede62aa66ba4b829be7c283993e3cf7
@ 0x7f4978f536db start_thread at ??:0
I20250624 07:55:12.061154 14043 consensus_queue.cc:1035] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f4971c8c71f clone at ??:0
I20250624 07:55:12.067128 13796 heartbeater.cc:499] Master 127.12.180.61:46565 was elected leader, sending a full tablet report...
I20250624 07:55:12.075240 14003 consensus_queue.cc:1035] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:12.078399 14182 mvcc.cc:204] Tried to move back new op lower bound from 7171079012217434112 to 7171079005202792448. Current Snapshot: MvccSnapshot[applied={T|T < 7171079012217434112 or (T in {7171079012217434112})}]
I20250624 07:55:12.219560 14025 consensus_queue.cc:1035] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:12.225044 14003 consensus_queue.cc:1035] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250624 07:55:12.241339 14199 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:12.040633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.12.180.3:33179)
0624 07:55:12.067191 (+ 26558us) server_negotiation.cc:206] Beginning negotiation
0624 07:55:12.067196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0624 07:55:12.241031 (+173835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44570: BlockingRecv error: recv got EOF from 127.0.0.1:44570 (error 108)
Metrics: {"server-negotiator.queue_time_us":26458,"thread_start_us":26554,"threads_started":1}
W20250624 07:55:12.241654 13684 connection.cc:537] client connection to 127.12.180.61:46565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:12.241971 14197 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:12.039923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.12.180.1:39271)
0624 07:55:12.068295 (+ 28372us) server_negotiation.cc:206] Beginning negotiation
0624 07:55:12.068301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0624 07:55:12.068318 (+ 17us) server_negotiation.cc:407] Connection header received
0624 07:55:12.068377 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0624 07:55:12.068383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0624 07:55:12.068444 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0624 07:55:12.068534 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0624 07:55:12.241014 (+172480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37446: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":28275,"thread_start_us":29046,"threads_started":1}
W20250624 07:55:12.244010 14196 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:12.039164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.12.180.2:43137)
0624 07:55:12.067525 (+ 28361us) server_negotiation.cc:206] Beginning negotiation
0624 07:55:12.067531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0624 07:55:12.067549 (+ 18us) server_negotiation.cc:407] Connection header received
0624 07:55:12.067614 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0624 07:55:12.067620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0624 07:55:12.067716 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0624 07:55:12.067832 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0624 07:55:12.243123 (+175291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34370: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":28286,"thread_start_us":28189,"threads_started":1}
W20250624 07:55:12.245036 13547 connection.cc:537] client connection to 127.12.180.61:46565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 07:55:12.282469 14003 consensus_queue.cc:1035] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:12.287382 14043 consensus_queue.cc:1035] T 31617090f65045f08d94b793ff525275 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:12.487411 14025 consensus_queue.cc:1035] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:12.493839 14025 consensus_queue.cc:1035] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:12.535586 14222 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:12.535799 14222 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:12.537395 14222 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:12.538475 14088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:12.539707 14087 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:12.539970 14088 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
I20250624 07:55:12.540130 14088 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:12.540496 14222 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250624 07:55:12.609087 14140 consensus_queue.cc:1035] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:12.617075 14228 consensus_queue.cc:1035] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:13.009737 14228 consensus_queue.cc:1035] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:13.014384 14140 consensus_queue.cc:1035] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:13.200659 14237 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:13.200845 14237 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:13.201489 14237 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:13.204684 14088 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:13.204856 14087 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:13.205000 14087 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:13.205255 14237 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250624 07:55:13.700373 13008 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu pid 14014
I20250624 07:55:13.710580 14131 generic_service.cc:196] Checking for leaks (request via RPC)
I20250624 07:55:13.809959 13008 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu with pid 14014
I20250624 07:55:14.749992 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--webserver_interface=127.12.180.60
--webserver_port=42595
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250624 07:55:15.080937 14271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:15.081277 14271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:15.081343 14271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:15.089893 14271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:15.090027 14271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:15.090152 14271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:15.090309 14271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:15.100184 14271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.180.60
--webserver_port=42595
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:15.101873 14271 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:15.104141 14271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:15.114535 14284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:15.114579 14286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:15.114498 14283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:15.116071 14271 server_base.cc:1048] running on GCE node
I20250624 07:55:15.116840 14271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:15.118566 14271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:15.119782 14271 hybrid_clock.cc:648] HybridClock initialized: now 1750751715119740 us; error 45 us; skew 500 ppm
I20250624 07:55:15.120146 14271 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:15.122483 14271 webserver.cc:469] Webserver started at http://127.12.180.60:42595/ using document root <none> and password file <none>
I20250624 07:55:15.123248 14271 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:15.123355 14271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:15.129618 14271 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20250624 07:55:15.132469 14293 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:15.133750 14271 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250624 07:55:15.133912 14271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
uuid: "632ba412a681447e93af32d7fdf0691d"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:15.134647 14271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:15.149219 14271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:15.150082 14271 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:15.150403 14271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:15.182725 14271 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.60:33449
I20250624 07:55:15.182747 14347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.60:33449 every 8 connection(s)
I20250624 07:55:15.184298 14271 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
I20250624 07:55:15.190054 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 14271
I20250624 07:55:15.190933 14348 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:15.194752 14348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap starting.
I20250624 07:55:15.206534 14348 log.cc:826] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:15.230845 14348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=24 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:55:15.231405 14348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap complete.
I20250624 07:55:15.236810 14348 raft_consensus.cc:357] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:15.237505 14348 raft_consensus.cc:738] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Initialized, Role: FOLLOWER
I20250624 07:55:15.238134 14348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:15.238968 14352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:15.239295 14352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:15.240005 14348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:15.252270 14366 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:15.252385 14366 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: loading cluster ID for follower catalog manager: success
I20250624 07:55:15.249228 14271 master_runner.cc:186] Error getting master registration for 127.12.180.62:44637: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111),
I20250624 07:55:15.254781 14271 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:15.261818 14366 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: acquiring CA information for follower catalog manager: success
I20250624 07:55:15.263979 14366 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:15.500517 13794 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:15.504215 14313 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:33907; Asking this server to re-register.
I20250624 07:55:15.504920 13794 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:15.506697 14313 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:15.554826 13657 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:15.556530 14313 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:59909; Asking this server to re-register.
I20250624 07:55:15.557123 13657 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:15.558581 14313 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:15.675776 13930 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:15.677353 14313 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:60629; Asking this server to re-register.
I20250624 07:55:15.677899 13930 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:15.679404 14313 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:15.766589 14377 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:15.766844 14377 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:15.768548 14377 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:15.769482 14297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:15.770840 14296 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:15.771004 14297 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
I20250624 07:55:15.771104 14296 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:15.771479 14377 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250624 07:55:16.337503 14383 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:16.337684 14383 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:16.338376 14383 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:16.340193 14297 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:16.340427 14296 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:16.340528 14296 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:16.340778 14383 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250624 07:55:17.201785 14398 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:17.201982 14398 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:17.202600 14398 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:17.204455 14297 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:17.204643 14296 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:17.204751 14296 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:17.204982 14398 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250624 07:55:18.066793 14412 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:18.066992 14412 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:18.067653 14412 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:18.069870 14297 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:18.070061 14296 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:18.070211 14297 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88
I20250624 07:55:18.070502 14412 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250624 07:55:18.368816 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--webserver_interface=127.12.180.62
--webserver_port=45479
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250624 07:55:18.686615 14419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:18.686946 14419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:18.687055 14419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:18.694669 14419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:18.694757 14419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:18.694821 14419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:18.694890 14419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:18.703830 14419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.180.62
--webserver_port=45479
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:18.705404 14419 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:18.707234 14419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:18.717298 14430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:18.717273 14431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:18.717522 14433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:18.717945 14419 server_base.cc:1048] running on GCE node
I20250624 07:55:18.718704 14419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:18.720078 14419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:18.721271 14419 hybrid_clock.cc:648] HybridClock initialized: now 1750751718721233 us; error 27 us; skew 500 ppm
I20250624 07:55:18.721585 14419 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:18.729655 14419 webserver.cc:469] Webserver started at http://127.12.180.62:45479/ using document root <none> and password file <none>
I20250624 07:55:18.730475 14419 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:18.730595 14419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:18.735713 14419 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250624 07:55:18.738529 14439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:18.739713 14419 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250624 07:55:18.739886 14419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
uuid: "588e956d4ac040deb3c06e57bd3c6b88"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:18.740651 14419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:18.762774 14419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:18.763752 14419 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:18.764020 14419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:18.782816 14419 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.62:44637
I20250624 07:55:18.782837 14490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.62:44637 every 8 connection(s)
I20250624 07:55:18.784349 14419 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
I20250624 07:55:18.789268 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 14419
I20250624 07:55:18.791004 14491 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:18.794523 14491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap starting.
I20250624 07:55:18.805936 14491 log.cc:826] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:18.834270 14491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=23 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:55:18.834791 14491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap complete.
I20250624 07:55:18.839965 14491 raft_consensus.cc:357] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:18.840652 14491 raft_consensus.cc:738] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588e956d4ac040deb3c06e57bd3c6b88, State: Initialized, Role: FOLLOWER
I20250624 07:55:18.841369 14491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:18.842223 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:18.842491 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:18.843182 14491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:18.862066 14419 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:18.865516 14419 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:18.869095 14509 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:18.869197 14509 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: loading cluster ID for follower catalog manager: success
I20250624 07:55:18.872463 14509 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: acquiring CA information for follower catalog manager: success
I20250624 07:55:18.874905 14509 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:19.012415 13795 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:19.017323 14456 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:47337; Asking this server to re-register.
I20250624 07:55:19.018260 13795 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:19.020319 14456 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:19.064402 13658 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:19.066326 14456 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:44825; Asking this server to re-register.
I20250624 07:55:19.066964 13658 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:19.068305 14456 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:19.185761 13931 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:19.187631 14456 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:36065; Asking this server to re-register.
I20250624 07:55:19.188225 13931 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:19.189455 14456 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:19.293064 14495 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:19.293340 14495 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:19.295249 14495 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:19.296025 14442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:19.297559 14442 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:19.301673 14323 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "632ba412a681447e93af32d7fdf0691d" is_pre_election: true
I20250624 07:55:19.302101 14323 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 588e956d4ac040deb3c06e57bd3c6b88 in current term 3: Already voted for candidate 52afd679ffb241a3a9022cc850c03493 in this term.
I20250624 07:55:19.302803 14440 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:19.303171 14495 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 2 FOLLOWER]: Advancing to term 3
I20250624 07:55:19.306882 14495 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250624 07:55:19.938591 14524 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:19.938778 14524 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:19.939390 14524 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:19.939997 14323 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 4 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "632ba412a681447e93af32d7fdf0691d" is_pre_election: true
I20250624 07:55:19.940299 14323 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 588e956d4ac040deb3c06e57bd3c6b88 for term 4 because replica has last-logged OpId of term: 3 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 32.
W20250624 07:55:19.941319 14442 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:19.941412 14442 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:19.941622 14524 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250624 07:55:20.178617 14531 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:20.178815 14531 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:20.179662 14531 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:20.188014 14296 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:20.194520 14466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
I20250624 07:55:20.195098 14466 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 3.
I20250624 07:55:20.195818 14297 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493
I20250624 07:55:20.198061 14531 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250624 07:55:20.198213 14531 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:20.198369 14531 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 3 FOLLOWER]: Advancing to term 4
I20250624 07:55:20.203915 14531 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:20.204550 14531 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 election: Requested vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
I20250624 07:55:20.205096 14466 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88"
I20250624 07:55:20.205308 14466 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 3 FOLLOWER]: Advancing to term 4
I20250624 07:55:20.208232 14466 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 4.
I20250624 07:55:20.209409 14297 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters:
I20250624 07:55:20.209684 14531 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Leader election won for term 4
W20250624 07:55:20.209753 14296 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:20.210129 14531 raft_consensus.cc:695] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 LEADER]: Becoming Leader. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Running, Role: LEADER
I20250624 07:55:20.210582 14531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 3.35, Last appended by leader: 35, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:20.213591 14537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 4 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:20.213903 14537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:20.214634 14539 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:20.218499 14539 catalog_manager.cc:671] Loaded metadata for table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:20.219098 14539 catalog_manager.cc:671] Loaded metadata for table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:20.219566 14539 catalog_manager.cc:671] Loaded metadata for table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:20.220036 14539 catalog_manager.cc:671] Loaded metadata for table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:20.220470 14539 catalog_manager.cc:671] Loaded metadata for table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:20.220935 14539 catalog_manager.cc:671] Loaded metadata for table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:20.221376 14539 catalog_manager.cc:671] Loaded metadata for table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:20.221804 14539 catalog_manager.cc:671] Loaded metadata for table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:20.222299 14539 catalog_manager.cc:671] Loaded metadata for table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:20.225031 14539 tablet_loader.cc:96] loaded metadata for tablet 09b765ab702f4817879ab30ebb558336 (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72])
I20250624 07:55:20.225592 14539 tablet_loader.cc:96] loaded metadata for tablet 2b11e2bf606e4e94a579bcf49502d94b (table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d])
I20250624 07:55:20.226076 14539 tablet_loader.cc:96] loaded metadata for tablet 31617090f65045f08d94b793ff525275 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2])
I20250624 07:55:20.226800 14539 tablet_loader.cc:96] loaded metadata for tablet 5daee63b211a4b099533045210d213c8 (table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62])
I20250624 07:55:20.227224 14539 tablet_loader.cc:96] loaded metadata for tablet 6d7a465332db49218a4bb64803056fb3 (table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939])
I20250624 07:55:20.227875 14539 tablet_loader.cc:96] loaded metadata for tablet 82f70c673a9749be9fc70112188be360 (table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e])
I20250624 07:55:20.228245 14539 tablet_loader.cc:96] loaded metadata for tablet 8ede62aa66ba4b829be7c283993e3cf7 (table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b])
I20250624 07:55:20.228881 14539 tablet_loader.cc:96] loaded metadata for tablet c95dc3a9620745c6a7020ca7434e2c1e (table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e])
I20250624 07:55:20.229323 14539 tablet_loader.cc:96] loaded metadata for tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542])
I20250624 07:55:20.229619 14539 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:20.230603 14539 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:20.230672 14539 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:20.231863 14539 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:20.232676 14539 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Loaded TSK: 0
I20250624 07:55:20.233501 14539 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:20.234544 14539 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:20.235419 14539 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
F20250624 07:55:20.253145 14365 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7efef5f7edcd google::LogMessage::Fail() at ??:0
@ 0x7efef5f82b93 google::LogMessage::SendToLog() at ??:0
@ 0x7efef5f7e7cc google::LogMessage::Flush() at ??:0
@ 0x7efef5f7ea11 google::LogMessage::~LogMessage() at ??:0
I20250624 07:55:20.260414 14271 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:20.260705 14271 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
@ 0x7eff0c37f6d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7eff0c37df81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7eff0c385bfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250624 07:55:20.275184 14466 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Refusing update from remote peer 632ba412a681447e93af32d7fdf0691d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
@ 0x7eff0c385a79 std::_Function_handler<>::_M_invoke() at ??:0
I20250624 07:55:20.276139 14537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Connected to new peer: Peer: permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 32, Time since last communication: 0.000s
W20250624 07:55:20.277420 14296 consensus_peers.cc:487] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d -> Peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Couldn't send request to peer 52afd679ffb241a3a9022cc850c03493. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0xa7573d std::function<>::operator()() at ??:?
@ 0x7efef8cd84fe kudu::Thread::SuperviseThread() at ??:0
@ 0x7efefc0f66db start_thread at ??:0
@ 0x7efef4e2f71f clone at ??:0
I20250624 07:55:20.288695 14548 mvcc.cc:204] Tried to move back new op lower bound from 7171079012475805696 to 7171079011943358464. Current Snapshot: MvccSnapshot[applied={T|T < 7171079012475805696 or (T in {7171079012475805696})}]
I20250624 07:55:20.288962 14524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 4 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:20.289134 14524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:20.289368 14550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:20.289512 14550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
W20250624 07:55:20.446218 14443 connection.cc:537] server connection from 127.0.0.1:57804 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:20.446668 13937 connection.cc:537] client connection to 127.12.180.60:33449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 07:55:20.780488 14550 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:20.780694 14550 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:20.781347 14550 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:20.783558 14442 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:20.783776 14440 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:20.783943 14440 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:20.784224 14550 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250624 07:55:20.945448 13008 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu pid 14419
I20250624 07:55:20.952224 14486 generic_service.cc:196] Checking for leaks (request via RPC)
I20250624 07:55:21.053045 13008 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu with pid 14419
I20250624 07:55:21.970300 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--webserver_interface=127.12.180.62
--webserver_port=45479
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250624 07:55:22.293412 14571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:22.293859 14571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:22.293994 14571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:22.301777 14571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:22.301895 14571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:22.301949 14571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:22.301987 14571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:22.311847 14571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.180.62
--webserver_port=45479
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:22.313575 14571 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:22.315677 14571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:22.326206 14577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:22.326467 14579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:22.327018 14576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:22.327314 14571 server_base.cc:1048] running on GCE node
I20250624 07:55:22.328130 14571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:22.329684 14571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:22.330888 14571 hybrid_clock.cc:648] HybridClock initialized: now 1750751722330830 us; error 50 us; skew 500 ppm
I20250624 07:55:22.331238 14571 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:22.333472 14571 webserver.cc:469] Webserver started at http://127.12.180.62:45479/ using document root <none> and password file <none>
I20250624 07:55:22.334268 14571 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:22.334375 14571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:22.339704 14571 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20250624 07:55:22.342825 14585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:22.344254 14571 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250624 07:55:22.344425 14571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
uuid: "588e956d4ac040deb3c06e57bd3c6b88"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:22.345165 14571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:22.368515 14571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:22.369514 14571 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:22.369807 14571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:22.390195 14571 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.62:44637
I20250624 07:55:22.390269 14636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.62:44637 every 8 connection(s)
I20250624 07:55:22.392127 14571 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
I20250624 07:55:22.398902 14637 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:22.400071 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 14571
I20250624 07:55:22.403456 14637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap starting.
I20250624 07:55:22.419967 14637 log.cc:826] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:22.447048 14637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=35 ignored=0} inserts{seen=24 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250624 07:55:22.447724 14637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap complete.
I20250624 07:55:22.452698 14637 raft_consensus.cc:357] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:22.455291 14637 raft_consensus.cc:738] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588e956d4ac040deb3c06e57bd3c6b88, State: Initialized, Role: FOLLOWER
I20250624 07:55:22.455999 14637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 4.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:22.456665 14643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:22.456880 14643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:22.457680 14637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:22.468312 14657 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:22.468410 14657 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: loading cluster ID for follower catalog manager: success
I20250624 07:55:22.483378 14657 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: acquiring CA information for follower catalog manager: success
I20250624 07:55:22.486266 14657 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:22.481863 14571 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:22.486860 14571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:22.563804 13795 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:22.566833 14601 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:39509; Asking this server to re-register.
I20250624 07:55:22.567974 13795 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:22.569677 14601 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:22.585120 13658 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:22.586463 14601 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:36395; Asking this server to re-register.
I20250624 07:55:22.587042 13658 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:22.588359 14601 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:22.708704 13931 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:22.710063 14601 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:57473; Asking this server to re-register.
I20250624 07:55:22.710659 13931 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:22.711881 14601 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:23.070427 14664 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:23.070662 14664 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:23.072696 14664 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:23.073448 14588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:23.074424 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:23.074595 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:23.074738 14588 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:23.075078 14664 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250624 07:55:23.756263 14669 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:23.756511 14669 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:23.757196 14669 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:23.759351 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:23.759517 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:23.759660 14588 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:23.759928 14669 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250624 07:55:24.784256 14675 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:24.784500 14675 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:24.785182 14675 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:24.787147 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:24.787390 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:24.787493 14586 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:24.787739 14675 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250624 07:55:25.641050 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--webserver_interface=127.12.180.60
--webserver_port=42595
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250624 07:55:25.659672 14681 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:25.659873 14681 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:25.660674 14681 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:25.662942 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:25.663144 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:25.663245 14586 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:25.663538 14681 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250624 07:55:25.993077 14680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:25.993486 14680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:25.993616 14680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:26.002121 14680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:26.002342 14680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:26.002465 14680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:26.002552 14680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:26.012759 14680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.180.60
--webserver_port=42595
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:26.014603 14680 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:26.016515 14680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:26.028410 14690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:26.028401 14689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:26.028650 14692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:26.029158 14680 server_base.cc:1048] running on GCE node
I20250624 07:55:26.030256 14680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:26.031946 14680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:26.033193 14680 hybrid_clock.cc:648] HybridClock initialized: now 1750751726033140 us; error 51 us; skew 500 ppm
I20250624 07:55:26.033609 14680 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:26.040423 14680 webserver.cc:469] Webserver started at http://127.12.180.60:42595/ using document root <none> and password file <none>
I20250624 07:55:26.041352 14680 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:26.041467 14680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:26.047561 14680 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250624 07:55:26.050953 14698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:26.052481 14680 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250624 07:55:26.052681 14680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
uuid: "632ba412a681447e93af32d7fdf0691d"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:26.053484 14680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:26.079705 14680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:26.080644 14680 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:26.080929 14680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:26.102443 14680 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.60:33449
I20250624 07:55:26.102468 14749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.60:33449 every 8 connection(s)
I20250624 07:55:26.104210 14680 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
I20250624 07:55:26.111591 14750 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:26.112084 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 14680
I20250624 07:55:26.116456 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap starting.
I20250624 07:55:26.131423 14750 log.cc:826] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:26.161612 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=35 ignored=0} inserts{seen=24 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250624 07:55:26.162353 14750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap complete.
I20250624 07:55:26.168526 14750 raft_consensus.cc:357] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:26.171376 14750 raft_consensus.cc:738] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Initialized, Role: FOLLOWER
I20250624 07:55:26.172128 14750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 4.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:26.172940 14755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:26.173311 14755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:26.174332 14750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:26.201640 14769 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:26.201749 14769 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: loading cluster ID for follower catalog manager: success
I20250624 07:55:26.206662 14769 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: acquiring CA information for follower catalog manager: success
I20250624 07:55:26.201591 14680 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:26.207490 14680 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:26.209668 14769 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:26.219081 13930 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:26.221928 14715 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:48001; Asking this server to re-register.
I20250624 07:55:26.222957 13930 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:26.224992 14715 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:26.548563 13794 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:26.550230 14715 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:37939; Asking this server to re-register.
I20250624 07:55:26.550941 13794 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:26.552579 14715 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:26.597038 13657 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:26.598613 14715 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:34365; Asking this server to re-register.
I20250624 07:55:26.599222 13657 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:26.600611 14715 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:26.630456 14755 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:26.630713 14755 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:26.632673 14755 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
W20250624 07:55:26.633463 14701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:26.633450 14612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 5 candidate_status { last_received { term: 4 index: 37 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
I20250624 07:55:26.633869 14612 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 4.
I20250624 07:55:26.634546 14702 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters:
I20250624 07:55:26.634927 14755 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250624 07:55:26.635056 14755 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:26.635223 14755 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 4 FOLLOWER]: Advancing to term 5
W20250624 07:55:26.635430 14701 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:26.638944 14755 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:26.639693 14755 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 5 election: Requested vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
I20250624 07:55:26.640254 14612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 5 candidate_status { last_received { term: 4 index: 37 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88"
I20250624 07:55:26.640486 14612 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 4 FOLLOWER]: Advancing to term 5
W20250624 07:55:26.641125 14701 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:26.644071 14612 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 5.
I20250624 07:55:26.644640 14702 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493
I20250624 07:55:26.644999 14755 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Leader election won for term 5
I20250624 07:55:26.645413 14755 raft_consensus.cc:695] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 LEADER]: Becoming Leader. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Running, Role: LEADER
I20250624 07:55:26.645879 14755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.37, Last appended by leader: 37, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:26.648753 14776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 5 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:26.649065 14776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:27.245930 14612 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Refusing update from remote peer 632ba412a681447e93af32d7fdf0691d: Log matching property violated. Preceding OpId in replica: term: 4 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20250624 07:55:27.246934 14781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Connected to new peer: Peer: permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 35, Time since last communication: 0.000s
I20250624 07:55:27.250861 14782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 5 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:27.251212 14782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
W20250624 07:55:27.252636 14701 consensus_peers.cc:487] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d -> Peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Couldn't send request to peer 52afd679ffb241a3a9022cc850c03493. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:55:27.254840 14781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:27.255115 14781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:27.255514 14785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:27.255864 14785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:27.255853 14782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:27.255820 14792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:27.256067 14782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:27.256247 14792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:27.295436 14778 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:27.298079 14778 catalog_manager.cc:671] Loaded metadata for table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:27.298682 14778 catalog_manager.cc:671] Loaded metadata for table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:27.299074 14778 catalog_manager.cc:671] Loaded metadata for table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:27.299427 14778 catalog_manager.cc:671] Loaded metadata for table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:27.299758 14778 catalog_manager.cc:671] Loaded metadata for table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:27.300091 14778 catalog_manager.cc:671] Loaded metadata for table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:27.300433 14778 catalog_manager.cc:671] Loaded metadata for table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:27.300781 14778 catalog_manager.cc:671] Loaded metadata for table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:27.301147 14778 catalog_manager.cc:671] Loaded metadata for table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:27.302992 14778 tablet_loader.cc:96] loaded metadata for tablet 09b765ab702f4817879ab30ebb558336 (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72])
I20250624 07:55:27.303390 14778 tablet_loader.cc:96] loaded metadata for tablet 2b11e2bf606e4e94a579bcf49502d94b (table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d])
I20250624 07:55:27.303756 14778 tablet_loader.cc:96] loaded metadata for tablet 31617090f65045f08d94b793ff525275 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2])
I20250624 07:55:27.304221 14778 tablet_loader.cc:96] loaded metadata for tablet 5daee63b211a4b099533045210d213c8 (table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62])
I20250624 07:55:27.304517 14778 tablet_loader.cc:96] loaded metadata for tablet 6d7a465332db49218a4bb64803056fb3 (table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939])
I20250624 07:55:27.304991 14778 tablet_loader.cc:96] loaded metadata for tablet 82f70c673a9749be9fc70112188be360 (table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e])
I20250624 07:55:27.305289 14778 tablet_loader.cc:96] loaded metadata for tablet 8ede62aa66ba4b829be7c283993e3cf7 (table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b])
I20250624 07:55:27.305720 14778 tablet_loader.cc:96] loaded metadata for tablet c95dc3a9620745c6a7020ca7434e2c1e (table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e])
I20250624 07:55:27.306006 14778 tablet_loader.cc:96] loaded metadata for tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542])
I20250624 07:55:27.306320 14778 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:27.307399 14778 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:27.307483 14778 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:27.308804 14778 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:27.309810 14778 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Loaded TSK: 0
I20250624 07:55:27.310839 14778 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:27.311833 14778 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:27.312708 14778 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250624 07:55:27.443961 14715 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59142:
name: "default.Table_26071aa0d9c8459b8f1d9583dedee21a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:27.444514 14710 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59142:
name: "default.Table_50b9eb18e89f4d0a9d1b64950157ffe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:27.444881 14714 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59142:
table { table_name: "default.Table_2974357596ca4893b46d7199b1367e2f" } new_table_name: "default.Table_77a9deb86184406588f30ae47cdb52d6" modify_external_catalogs: true
I20250624 07:55:27.445240 14713 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59142:
table { table_name: "default.Table_0047240c5fdf4b85bc2b5fda6bdcef32" } new_table_name: "default.Table_34f740d34f55471b82a8fd8be1423bbf" modify_external_catalogs: true
I20250624 07:55:27.445780 14708 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59142:
table { table_name: "default.Table_8a84bf94dbd94e65946d5e56b69172be" } new_table_name: "default.Table_d133c18e2676484dabe455c76b424d05" modify_external_catalogs: true
I20250624 07:55:27.445343 14707 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59142:
name: "default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:27.445644 14712 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59142:
table { table_name: "default.Table_fc2c06ca3ddc4d78baaced0d6345b281" } new_table_name: "default.Table_4b83c91d0994483c8da55d71dafb8eeb" modify_external_catalogs: true
I20250624 07:55:27.446676 14711 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59142:
table { table_name: "default.Table_ba78959886d2470285dacaf16007267d" } new_table_name: "default.Table_f6122adb52c04c12b5cad442b855c049" modify_external_catalogs: true
I20250624 07:55:27.446846 14709 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59142:
name: "default.Table_258e5344814340e49754e36a4631c5e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:27.445444 14706 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59142:
name: "default.Table_6c5708ec0a8a44f4bc2b8d669d041a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250624 07:55:27.452471 14768 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f134f929dcd google::LogMessage::Fail() at ??:0
@ 0x7f134f92db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f134f9297cc google::LogMessage::Flush() at ??:0
@ 0x7f134f929a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f1365d2a6d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f1365d28f81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f1365d30bfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f1365d30a79 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa7573d std::function<>::operator()() at ??:?
@ 0x7f13526834fe kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1355aa16db start_thread at ??:0
@ 0x7f134e7da71f clone at ??:0
I20250624 07:55:27.491068 14571 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:27.491273 14571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:55:27.661280 13545 connection.cc:537] client connection to 127.12.180.60:33449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:27.661291 13682 connection.cc:537] client connection to 127.12.180.60:33449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:27.662505 13937 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.180.60:33449)
W20250624 07:55:27.662787 13937 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.180.60:33449)
W20250624 07:55:27.663035 13937 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.180.60:33449)
W20250624 07:55:27.663842 13937 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.180.60:33449)
I20250624 07:55:27.753471 14792 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:27.753670 14792 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:27.754446 14792 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:27.756506 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:27.756776 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:27.756968 14586 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:27.757233 14792 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250624 07:55:28.463387 14814 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:28.463580 14814 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:28.464308 14814 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:28.465255 14588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250624 07:55:28.466571 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:28.466818 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:28.466955 14588 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:28.467223 14814 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250624 07:55:29.341240 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--webserver_interface=127.12.180.60
--webserver_port=42595
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250624 07:55:29.567169 14820 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:29.567354 14820 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:29.568074 14820 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:29.570097 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:29.570322 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:29.570468 14586 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:29.570755 14820 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250624 07:55:29.671798 14819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:29.672156 14819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:29.672222 14819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:29.680327 14819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:29.680467 14819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:29.680612 14819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:29.680667 14819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:29.690541 14819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.60:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.180.60
--webserver_port=42595
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:29.692361 14819 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:29.694346 14819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:29.705030 14832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:29.704972 14829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:29.704931 14830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:29.706353 14819 server_base.cc:1048] running on GCE node
I20250624 07:55:29.707156 14819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:29.708626 14819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:29.709825 14819 hybrid_clock.cc:648] HybridClock initialized: now 1750751729709756 us; error 61 us; skew 500 ppm
I20250624 07:55:29.710211 14819 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:29.712653 14819 webserver.cc:469] Webserver started at http://127.12.180.60:42595/ using document root <none> and password file <none>
I20250624 07:55:29.713482 14819 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:29.713593 14819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:29.719274 14819 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250624 07:55:29.722204 14838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:29.724052 14819 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250624 07:55:29.724241 14819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
uuid: "632ba412a681447e93af32d7fdf0691d"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:29.724943 14819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:29.741302 14819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:29.742151 14819 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:29.742506 14819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:29.761979 14819 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.60:33449
I20250624 07:55:29.762001 14889 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.60:33449 every 8 connection(s)
I20250624 07:55:29.763669 14819 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-2/data/info.pb
I20250624 07:55:29.769855 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 14819
I20250624 07:55:29.769891 14890 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:29.774286 14890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap starting.
I20250624 07:55:29.787161 14890 log.cc:826] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:29.816877 14890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=24 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:55:29.817507 14890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Bootstrap complete.
I20250624 07:55:29.822633 14890 raft_consensus.cc:357] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:29.823339 14890 raft_consensus.cc:738] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Initialized, Role: FOLLOWER
I20250624 07:55:29.823915 14890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 5.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:29.824707 14894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:29.825006 14894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:29.825783 14890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:29.838816 14908 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:29.839030 14908 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: loading cluster ID for follower catalog manager: success
I20250624 07:55:29.853502 14908 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: acquiring CA information for follower catalog manager: success
I20250624 07:55:29.853009 14819 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:29.858551 14819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:29.861830 14908 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:30.175544 13794 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:30.175616 13657 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:30.178463 14854 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:56537; Asking this server to re-register.
I20250624 07:55:30.178491 14855 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:56563; Asking this server to re-register.
I20250624 07:55:30.179270 13794 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:30.179456 13657 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:30.181106 14854 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:30.181175 14855 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:30.243189 13930 heartbeater.cc:344] Connected to a master server at 127.12.180.60:33449
I20250624 07:55:30.245940 14854 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:56303; Asking this server to re-register.
I20250624 07:55:30.246706 13930 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:30.248191 14854 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:30.338083 14914 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:30.338410 14914 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:30.340148 14914 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
I20250624 07:55:30.340745 14612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
W20250624 07:55:30.340809 14841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:30.340963 14612 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 5.
I20250624 07:55:30.341526 14842 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters:
I20250624 07:55:30.341881 14914 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250624 07:55:30.342007 14914 raft_consensus.cc:491] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:30.342119 14914 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 5 FOLLOWER]: Advancing to term 6
W20250624 07:55:30.342249 14841 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:30.345731 14914 raft_consensus.cc:513] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:30.346359 14914 leader_election.cc:290] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 6 election: Requested vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565)
I20250624 07:55:30.346794 14612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "632ba412a681447e93af32d7fdf0691d" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88"
I20250624 07:55:30.347005 14612 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 5 FOLLOWER]: Advancing to term 6
W20250624 07:55:30.348327 14841 leader_election.cc:336] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:30.349452 14612 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 632ba412a681447e93af32d7fdf0691d in term 6.
I20250624 07:55:30.349875 14842 leader_election.cc:304] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d; no voters: 52afd679ffb241a3a9022cc850c03493
I20250624 07:55:30.350204 14914 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 6 FOLLOWER]: Leader election won for term 6
I20250624 07:55:30.350600 14914 raft_consensus.cc:695] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [term 6 LEADER]: Becoming Leader. State: Replica: 632ba412a681447e93af32d7fdf0691d, State: Running, Role: LEADER
I20250624 07:55:30.351047 14914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 5.38, Last appended by leader: 38, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:30.353746 14916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 6 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:30.353976 14916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:30.354586 14918 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:30.356861 14918 catalog_manager.cc:671] Loaded metadata for table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:30.357221 14918 catalog_manager.cc:671] Loaded metadata for table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:30.357508 14918 catalog_manager.cc:671] Loaded metadata for table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:30.357779 14918 catalog_manager.cc:671] Loaded metadata for table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:30.358039 14918 catalog_manager.cc:671] Loaded metadata for table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:30.358377 14918 catalog_manager.cc:671] Loaded metadata for table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:30.358685 14918 catalog_manager.cc:671] Loaded metadata for table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:30.358961 14918 catalog_manager.cc:671] Loaded metadata for table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:30.359222 14918 catalog_manager.cc:671] Loaded metadata for table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:30.360682 14918 tablet_loader.cc:96] loaded metadata for tablet 09b765ab702f4817879ab30ebb558336 (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72])
I20250624 07:55:30.361075 14918 tablet_loader.cc:96] loaded metadata for tablet 2b11e2bf606e4e94a579bcf49502d94b (table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d])
I20250624 07:55:30.361356 14918 tablet_loader.cc:96] loaded metadata for tablet 31617090f65045f08d94b793ff525275 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2])
I20250624 07:55:30.361712 14918 tablet_loader.cc:96] loaded metadata for tablet 5daee63b211a4b099533045210d213c8 (table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62])
I20250624 07:55:30.361965 14918 tablet_loader.cc:96] loaded metadata for tablet 6d7a465332db49218a4bb64803056fb3 (table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939])
I20250624 07:55:30.362388 14918 tablet_loader.cc:96] loaded metadata for tablet 82f70c673a9749be9fc70112188be360 (table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e])
I20250624 07:55:30.362648 14918 tablet_loader.cc:96] loaded metadata for tablet 8ede62aa66ba4b829be7c283993e3cf7 (table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b])
I20250624 07:55:30.363050 14918 tablet_loader.cc:96] loaded metadata for tablet c95dc3a9620745c6a7020ca7434e2c1e (table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e])
I20250624 07:55:30.363286 14918 tablet_loader.cc:96] loaded metadata for tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542])
I20250624 07:55:30.363477 14918 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:30.364343 14918 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:30.364408 14918 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:30.365533 14918 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:30.366369 14918 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Loaded TSK: 0
I20250624 07:55:30.367166 14918 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:30.368028 14918 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:30.368783 14918 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250624 07:55:30.425837 14854 master_service.cc:948] ReplaceTablet: received request to replace tablet 2b11e2bf606e4e94a579bcf49502d94b from {username='slave'} at 127.0.0.1:52222
I20250624 07:55:30.426239 14855 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52222:
name: "default.Table_ec6cab3afe9e48f3a38fa9c0c2ca3922"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:30.426606 14853 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52222:
table { table_name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe" } modify_external_catalogs: true
I20250624 07:55:30.426947 14853 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52222:
table { table_name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe" } modify_external_catalogs: true
I20250624 07:55:30.427101 14852 master_service.cc:948] ReplaceTablet: received request to replace tablet c95dc3a9620745c6a7020ca7434e2c1e from {username='slave'} at 127.0.0.1:52222
I20250624 07:55:30.426971 14851 master_service.cc:948] ReplaceTablet: received request to replace tablet 09b765ab702f4817879ab30ebb558336 from {username='slave'} at 127.0.0.1:52222
I20250624 07:55:30.429484 14850 master_service.cc:948] ReplaceTablet: received request to replace tablet 82f70c673a9749be9fc70112188be360 from {username='slave'} at 127.0.0.1:52222
I20250624 07:55:30.430362 14849 master_service.cc:948] ReplaceTablet: received request to replace tablet 6d7a465332db49218a4bb64803056fb3 from {username='slave'} at 127.0.0.1:52222
I20250624 07:55:30.430666 14846 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52222:
table { table_name: "default.Table_ba78959886d2470285dacaf16007267d" } new_table_name: "default.Table_f6122adb52c04c12b5cad442b855c049" modify_external_catalogs: true
I20250624 07:55:30.430775 14848 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52222:
name: "default.Table_c316e293c95c47eda371175ca862fe0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:30.430040 14847 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52222:
name: "default.Table_50b9eb18e89f4d0a9d1b64950157ffe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250624 07:55:30.436848 14907 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fb73ef38dcd google::LogMessage::Fail() at ??:0
@ 0x7fb73ef3cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb73ef387cc google::LogMessage::Flush() at ??:0
I20250624 07:55:30.443397 14612 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Refusing update from remote peer 632ba412a681447e93af32d7fdf0691d: Log matching property violated. Preceding OpId in replica: term: 5 index: 38. Preceding OpId from leader: term: 6 index: 41. (index mismatch)
W20250624 07:55:30.443809 14841 consensus_peers.cc:487] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d -> Peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Couldn't send request to peer 52afd679ffb241a3a9022cc850c03493. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fb73ef38a11 google::LogMessage::~LogMessage() at ??:0
I20250624 07:55:30.444324 14916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [LEADER]: Connected to new peer: Peer: permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
@ 0x7fb7553396d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250624 07:55:30.456338 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632ba412a681447e93af32d7fdf0691d. Latest consensus state: current_term: 6 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:30.456573 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:30.458066 14920 mvcc.cc:204] Tried to move back new op lower bound from 7171079087873208320 to 7171079087521456128. Current Snapshot: MvccSnapshot[applied={T|T < 7171079087873208320}]
@ 0x7fb755337f81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250624 07:55:30.463766 14851 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Sending DeleteTablet for 3 replicas of tablet 09b765ab702f4817879ab30ebb558336
@ 0x7fb75533fbfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250624 07:55:30.464239 14852 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Sending DeleteTablet for 3 replicas of tablet c95dc3a9620745c6a7020ca7434e2c1e
I20250624 07:55:30.466127 14852 catalog_manager.cc:6652] ReplaceTablet: tablet c95dc3a9620745c6a7020ca7434e2c1e deleted and replaced by tablet 901ae0683a114638a6a075cf24c5eabe
@ 0x7fb75533fa79 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa7573d std::function<>::operator()() at ??:?
I20250624 07:55:30.475239 14851 catalog_manager.cc:6652] ReplaceTablet: tablet 09b765ab702f4817879ab30ebb558336 deleted and replaced by tablet f5af61fc5261444b8544d72aa61d3082
I20250624 07:55:30.483137 14851 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52222:
table { table_name: "default.Table_0047240c5fdf4b85bc2b5fda6bdcef32" } new_table_name: "default.Table_34f740d34f55471b82a8fd8be1423bbf" modify_external_catalogs: true
I20250624 07:55:30.483812 14914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:30.484081 14914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d [sys.catalog]: This master's current role is: LEADER
@ 0x7fb741c924fe kudu::Thread::SuperviseThread() at ??:0
I20250624 07:55:30.488277 14852 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52222:
table { table_name: "default.Table_fc2c06ca3ddc4d78baaced0d6345b281" } new_table_name: "default.Table_4b83c91d0994483c8da55d71dafb8eeb" modify_external_catalogs: true
I20250624 07:55:30.489014 13863 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b765ab702f4817879ab30ebb558336 with delete_type TABLET_DATA_DELETED (replaced by tablet f5af61fc5261444b8544d72aa61d3082) from {username='slave'} at 127.0.0.1:58530
I20250624 07:55:30.489862 13865 tablet_service.cc:1515] Processing DeleteTablet for tablet c95dc3a9620745c6a7020ca7434e2c1e with delete_type TABLET_DATA_DELETED (replaced by tablet 901ae0683a114638a6a075cf24c5eabe) from {username='slave'} at 127.0.0.1:58530
I20250624 07:55:30.492555 13590 tablet_service.cc:1515] Processing DeleteTablet for tablet c95dc3a9620745c6a7020ca7434e2c1e with delete_type TABLET_DATA_DELETED (replaced by tablet 901ae0683a114638a6a075cf24c5eabe) from {username='slave'} at 127.0.0.1:40836
I20250624 07:55:30.492650 13591 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b765ab702f4817879ab30ebb558336 with delete_type TABLET_DATA_DELETED (replaced by tablet f5af61fc5261444b8544d72aa61d3082) from {username='slave'} at 127.0.0.1:40836
I20250624 07:55:30.496057 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "632ba412a681447e93af32d7fdf0691d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:30.496307 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:30.500561 14850 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Sending DeleteTablet for 3 replicas of tablet 82f70c673a9749be9fc70112188be360
I20250624 07:55:30.501149 14850 catalog_manager.cc:6652] ReplaceTablet: tablet 82f70c673a9749be9fc70112188be360 deleted and replaced by tablet 2fc78118d4fa41d0b4a1aabb92616335
I20250624 07:55:30.501819 14849 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Sending DeleteTablet for 3 replicas of tablet 6d7a465332db49218a4bb64803056fb3
I20250624 07:55:30.503585 13865 tablet_service.cc:1515] Processing DeleteTablet for tablet 82f70c673a9749be9fc70112188be360 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fc78118d4fa41d0b4a1aabb92616335) from {username='slave'} at 127.0.0.1:58530
I20250624 07:55:30.503708 13591 tablet_service.cc:1515] Processing DeleteTablet for tablet 82f70c673a9749be9fc70112188be360 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fc78118d4fa41d0b4a1aabb92616335) from {username='slave'} at 127.0.0.1:40836
I20250624 07:55:30.504089 13592 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d7a465332db49218a4bb64803056fb3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7e38715d1543c69f3ceee6fb74d54a) from {username='slave'} at 127.0.0.1:40836
I20250624 07:55:30.504196 13864 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d7a465332db49218a4bb64803056fb3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7e38715d1543c69f3ceee6fb74d54a) from {username='slave'} at 127.0.0.1:58530
I20250624 07:55:30.504371 14849 catalog_manager.cc:6652] ReplaceTablet: tablet 6d7a465332db49218a4bb64803056fb3 deleted and replaced by tablet 1d7e38715d1543c69f3ceee6fb74d54a
I20250624 07:55:30.506491 14854 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 632ba412a681447e93af32d7fdf0691d: Sending DeleteTablet for 3 replicas of tablet 2b11e2bf606e4e94a579bcf49502d94b
I20250624 07:55:30.507071 14854 catalog_manager.cc:6652] ReplaceTablet: tablet 2b11e2bf606e4e94a579bcf49502d94b deleted and replaced by tablet a7fc1a7fbc17491caa3b44e0a4db6590
I20250624 07:55:30.508282 13592 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b11e2bf606e4e94a579bcf49502d94b with delete_type TABLET_DATA_DELETED (replaced by tablet a7fc1a7fbc17491caa3b44e0a4db6590) from {username='slave'} at 127.0.0.1:40836
I20250624 07:55:30.508858 13864 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b11e2bf606e4e94a579bcf49502d94b with delete_type TABLET_DATA_DELETED (replaced by tablet a7fc1a7fbc17491caa3b44e0a4db6590) from {username='slave'} at 127.0.0.1:58530
@ 0x7fb7450b06db start_thread at ??:0
I20250624 07:55:30.510512 13728 tablet_service.cc:1515] Processing DeleteTablet for tablet c95dc3a9620745c6a7020ca7434e2c1e with delete_type TABLET_DATA_DELETED (replaced by tablet 901ae0683a114638a6a075cf24c5eabe) from {username='slave'} at 127.0.0.1:50930
I20250624 07:55:30.510912 13729 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b765ab702f4817879ab30ebb558336 with delete_type TABLET_DATA_DELETED (replaced by tablet f5af61fc5261444b8544d72aa61d3082) from {username='slave'} at 127.0.0.1:50930
I20250624 07:55:30.511170 13727 tablet_service.cc:1515] Processing DeleteTablet for tablet 82f70c673a9749be9fc70112188be360 with delete_type TABLET_DATA_DELETED (replaced by tablet 2fc78118d4fa41d0b4a1aabb92616335) from {username='slave'} at 127.0.0.1:50930
I20250624 07:55:30.511418 13726 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d7a465332db49218a4bb64803056fb3 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d7e38715d1543c69f3ceee6fb74d54a) from {username='slave'} at 127.0.0.1:50930
I20250624 07:55:30.511651 13725 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b11e2bf606e4e94a579bcf49502d94b with delete_type TABLET_DATA_DELETED (replaced by tablet a7fc1a7fbc17491caa3b44e0a4db6590) from {username='slave'} at 127.0.0.1:50930
@ 0x7fb73dde971f clone at ??:0
I20250624 07:55:30.513034 14939 tablet_replica.cc:331] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:30.513319 14939 raft_consensus.cc:2241] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.513651 14939 raft_consensus.cc:2270] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.513998 14938 tablet_replica.cc:331] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:30.514252 14938 raft_consensus.cc:2241] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.514425 14939 ts_tablet_manager.cc:1905] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.514473 14938 raft_consensus.cc:2270] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.515091 14938 ts_tablet_manager.cc:1905] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.521178 14943 tablet_replica.cc:331] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:30.521387 14943 raft_consensus.cc:2241] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Raft consensus shutting down.
I20250624 07:55:30.521657 14939 ts_tablet_manager.cc:1918] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.521796 14939 log.cc:1199] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/c95dc3a9620745c6a7020ca7434e2c1e
I20250624 07:55:30.522011 14943 raft_consensus.cc:2270] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.522270 14939 ts_tablet_manager.cc:1939] T c95dc3a9620745c6a7020ca7434e2c1e P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:30.522975 14943 ts_tablet_manager.cc:1905] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.523239 14939 tablet_replica.cc:331] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:30.523447 14939 raft_consensus.cc:2241] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.523599 14939 raft_consensus.cc:2270] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.523661 14938 ts_tablet_manager.cc:1918] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.523787 14938 log.cc:1199] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/09b765ab702f4817879ab30ebb558336
I20250624 07:55:30.524101 14939 ts_tablet_manager.cc:1905] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.524262 14938 ts_tablet_manager.cc:1939] T 09b765ab702f4817879ab30ebb558336 P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:30.525461 14938 tablet_replica.cc:331] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:30.525707 14938 raft_consensus.cc:2241] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.525880 14938 raft_consensus.cc:2270] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.526489 14938 ts_tablet_manager.cc:1905] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.529465 14943 ts_tablet_manager.cc:1918] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.529572 14943 log.cc:1199] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/c95dc3a9620745c6a7020ca7434e2c1e
I20250624 07:55:30.529959 14943 ts_tablet_manager.cc:1939] T c95dc3a9620745c6a7020ca7434e2c1e P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:30.530217 14939 ts_tablet_manager.cc:1918] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.530337 14939 log.cc:1199] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/09b765ab702f4817879ab30ebb558336
I20250624 07:55:30.530704 14939 ts_tablet_manager.cc:1939] T 09b765ab702f4817879ab30ebb558336 P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:30.531113 14943 tablet_replica.cc:331] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:30.531307 14943 raft_consensus.cc:2241] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Raft consensus shutting down.
I20250624 07:55:30.531572 14939 tablet_replica.cc:331] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:30.531617 14943 raft_consensus.cc:2270] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.531734 14939 raft_consensus.cc:2241] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.531908 14939 raft_consensus.cc:2270] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.532162 14943 ts_tablet_manager.cc:1905] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.532372 14939 ts_tablet_manager.cc:1905] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.533769 14938 ts_tablet_manager.cc:1918] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.533910 14938 log.cc:1199] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/c95dc3a9620745c6a7020ca7434e2c1e
I20250624 07:55:30.534345 14938 ts_tablet_manager.cc:1939] T c95dc3a9620745c6a7020ca7434e2c1e P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:30.535421 14938 tablet_replica.cc:331] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:30.535632 14938 raft_consensus.cc:2241] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.535818 14938 raft_consensus.cc:2270] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.536376 14938 ts_tablet_manager.cc:1905] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.539040 14939 ts_tablet_manager.cc:1918] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.539175 14939 log.cc:1199] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/82f70c673a9749be9fc70112188be360
I20250624 07:55:30.539604 14939 ts_tablet_manager.cc:1939] T 82f70c673a9749be9fc70112188be360 P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:30.540544 14939 tablet_replica.cc:331] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:30.540738 14939 raft_consensus.cc:2241] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.540987 14939 raft_consensus.cc:2270] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.541654 14939 ts_tablet_manager.cc:1905] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.543017 14938 ts_tablet_manager.cc:1918] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.543147 14938 log.cc:1199] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/82f70c673a9749be9fc70112188be360
I20250624 07:55:30.543542 14938 ts_tablet_manager.cc:1939] T 82f70c673a9749be9fc70112188be360 P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:30.544709 14938 tablet_replica.cc:331] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:30.544899 14938 raft_consensus.cc:2241] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.545069 14938 raft_consensus.cc:2270] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.545341 14943 ts_tablet_manager.cc:1918] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.545481 14943 log.cc:1199] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/09b765ab702f4817879ab30ebb558336
I20250624 07:55:30.545612 14938 ts_tablet_manager.cc:1905] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.545876 14943 ts_tablet_manager.cc:1939] T 09b765ab702f4817879ab30ebb558336 P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:30.547813 14943 tablet_replica.cc:331] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:30.547986 14943 raft_consensus.cc:2241] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Raft consensus shutting down.
I20250624 07:55:30.548303 14939 ts_tablet_manager.cc:1918] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.548348 14943 raft_consensus.cc:2270] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.548458 14939 log.cc:1199] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/6d7a465332db49218a4bb64803056fb3
I20250624 07:55:30.548866 14939 ts_tablet_manager.cc:1939] T 6d7a465332db49218a4bb64803056fb3 P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:30.548931 14943 ts_tablet_manager.cc:1905] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.549744 14939 tablet_replica.cc:331] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:30.549923 14939 raft_consensus.cc:2241] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.550153 14939 raft_consensus.cc:2270] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.550663 14939 ts_tablet_manager.cc:1905] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.552062 14938 ts_tablet_manager.cc:1918] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.552227 14938 log.cc:1199] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/6d7a465332db49218a4bb64803056fb3
I20250624 07:55:30.552662 14938 ts_tablet_manager.cc:1939] T 6d7a465332db49218a4bb64803056fb3 P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:30.553684 14938 tablet_replica.cc:331] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:30.553848 14938 raft_consensus.cc:2241] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 1 LEADER]: Raft consensus shutting down.
I20250624 07:55:30.554414 14938 raft_consensus.cc:2270] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.554972 14938 ts_tablet_manager.cc:1905] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.557235 14939 ts_tablet_manager.cc:1918] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.557394 14939 log.cc:1199] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/2b11e2bf606e4e94a579bcf49502d94b
I20250624 07:55:30.557989 14939 ts_tablet_manager.cc:1939] T 2b11e2bf606e4e94a579bcf49502d94b P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:30.559767 14943 ts_tablet_manager.cc:1918] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.559892 14943 log.cc:1199] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/82f70c673a9749be9fc70112188be360
I20250624 07:55:30.560390 14943 ts_tablet_manager.cc:1939] T 82f70c673a9749be9fc70112188be360 P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:30.561899 14938 ts_tablet_manager.cc:1918] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.562009 14938 log.cc:1199] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/2b11e2bf606e4e94a579bcf49502d94b
I20250624 07:55:30.562314 14943 tablet_replica.cc:331] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:30.562439 14938 ts_tablet_manager.cc:1939] T 2b11e2bf606e4e94a579bcf49502d94b P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:30.562534 14943 raft_consensus.cc:2241] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Raft consensus shutting down.
I20250624 07:55:30.562903 14943 raft_consensus.cc:2270] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.563525 14943 ts_tablet_manager.cc:1905] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.570539 14943 ts_tablet_manager.cc:1918] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.570678 14943 log.cc:1199] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/6d7a465332db49218a4bb64803056fb3
I20250624 07:55:30.571067 14943 ts_tablet_manager.cc:1939] T 6d7a465332db49218a4bb64803056fb3 P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:30.572134 14943 tablet_replica.cc:331] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:30.572325 14943 raft_consensus.cc:2241] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:30.572525 14943 raft_consensus.cc:2270] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:30.573158 14943 ts_tablet_manager.cc:1905] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:30.578405 14943 ts_tablet_manager.cc:1918] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:30.578505 14943 log.cc:1199] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/2b11e2bf606e4e94a579bcf49502d94b
I20250624 07:55:30.578830 14943 ts_tablet_manager.cc:1939] T 2b11e2bf606e4e94a579bcf49502d94b P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
W20250624 07:55:30.702701 13683 connection.cc:537] server connection from 127.0.0.1:50930 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:30.702759 13546 connection.cc:537] server connection from 127.0.0.1:40836 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:30.702761 13820 connection.cc:537] server connection from 127.0.0.1:58530 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:30.703200 13545 connection.cc:537] client connection to 127.12.180.60:33449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:30.703312 14587 connection.cc:537] server connection from 127.0.0.1:48388 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:30.703722 13937 connection.cc:537] client connection to 127.12.180.60:33449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:30.704018 13682 connection.cc:537] client connection to 127.12.180.60:33449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:30.704511 13937 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.180.60:33449)
W20250624 07:55:30.704898 13937 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.180.60:33449)
W20250624 07:55:30.705135 13937 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.180.60:33449)
W20250624 07:55:30.706452 13937 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.180.60:33449)
I20250624 07:55:31.001200 14952 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:31.001387 14952 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:31.002238 14952 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:31.004392 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:31.004601 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:31.004745 14586 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:31.005021 14952 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250624 07:55:31.751791 14960 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:31.751981 14960 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:31.752605 14960 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:31.756011 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:31.756194 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:31.756317 14586 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:31.756531 14960 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250624 07:55:32.487124 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.61:46565
--webserver_interface=127.12.180.61
--webserver_port=43475
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250624 07:55:32.494205 14571 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:32.494395 14571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:32.768013 14969 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:32.768217 14969 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:32.768879 14969 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:32.770989 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:32.771245 14588 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:32.771380 14588 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:32.771641 14969 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250624 07:55:32.802111 14967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:32.802561 14967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:32.802696 14967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:32.810883 14967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:32.810985 14967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:32.811082 14967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:32.811151 14967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:32.823282 14967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.61:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.180.61
--webserver_port=43475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:32.825546 14967 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:32.828166 14967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:32.839030 14976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:32.839030 14977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:32.839118 14979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:32.840334 14967 server_base.cc:1048] running on GCE node
I20250624 07:55:32.841074 14967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:32.843109 14967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:32.844372 14967 hybrid_clock.cc:648] HybridClock initialized: now 1750751732844282 us; error 78 us; skew 500 ppm
I20250624 07:55:32.844767 14967 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:32.856333 14967 webserver.cc:469] Webserver started at http://127.12.180.61:43475/ using document root <none> and password file <none>
I20250624 07:55:32.857172 14967 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:32.857291 14967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:32.862874 14967 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250624 07:55:32.865594 14985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:32.866914 14967 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250624 07:55:32.867071 14967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
uuid: "52afd679ffb241a3a9022cc850c03493"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:32.867699 14967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:32.882064 14967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:32.882927 14967 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:32.883196 14967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:32.903041 14967 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.61:46565
I20250624 07:55:32.903088 15036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.61:46565 every 8 connection(s)
I20250624 07:55:32.904699 14967 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
I20250624 07:55:32.909560 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 14967
I20250624 07:55:32.911201 15037 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:32.917104 15037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Bootstrap starting.
I20250624 07:55:32.929545 15037 log.cc:826] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:32.956009 15037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=24 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:55:32.956581 15037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Bootstrap complete.
I20250624 07:55:32.961772 15037 raft_consensus.cc:357] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:32.962492 15037 raft_consensus.cc:738] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52afd679ffb241a3a9022cc850c03493, State: Initialized, Role: FOLLOWER
I20250624 07:55:32.963089 15037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:32.963811 15042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:32.964030 15042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:32.964815 15037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:32.992604 15056 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:32.992722 15056 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: loading cluster ID for follower catalog manager: success
I20250624 07:55:33.002902 15056 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: acquiring CA information for follower catalog manager: success
I20250624 07:55:33.005824 15056 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:33.014761 14967 master_runner.cc:186] Error getting master registration for 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111),
I20250624 07:55:33.018435 14967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:33.046693 13796 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:33.046906 13796 heartbeater.cc:499] Master 127.12.180.61:46565 was elected leader, sending a full tablet report...
I20250624 07:55:33.050383 15002 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:47105; Asking this server to re-register.
I20250624 07:55:33.051203 13796 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:33.052989 15002 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:33.099893 13659 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:33.101503 15002 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:52149; Asking this server to re-register.
I20250624 07:55:33.102105 13659 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:33.103292 15002 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:33.229655 13932 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:33.231223 15002 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:46335; Asking this server to re-register.
I20250624 07:55:33.231808 13932 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:33.233073 15002 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:33.549952 15063 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:33.550307 15063 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:33.552021 15063 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:33.552558 14611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
I20250624 07:55:33.552845 14611 raft_consensus.cc:2366] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52afd679ffb241a3a9022cc850c03493 for earlier term 4. Current term is 6.
W20250624 07:55:33.553117 14986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:33.553591 14989 leader_election.cc:400] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52afd679ffb241a3a9022cc850c03493 for earlier term 4. Current term is 6.
I20250624 07:55:33.553719 14989 leader_election.cc:403] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250624 07:55:33.554042 15063 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 3 FOLLOWER]: Advancing to term 6
W20250624 07:55:33.554613 14986 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:33.557657 15063 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 6 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52afd679ffb241a3a9022cc850c03493 for earlier term 4. Current term is 6.
I20250624 07:55:33.662943 15066 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:33.663136 15066 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:33.663789 15066 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:33.664757 14586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250624 07:55:33.665977 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:33.669783 15012 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 7 candidate_status { last_received { term: 6 index: 45 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493" is_pre_election: true
I20250624 07:55:33.670287 15012 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588e956d4ac040deb3c06e57bd3c6b88 in term 6.
I20250624 07:55:33.670799 14588 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters: 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:33.671090 15066 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250624 07:55:33.671221 15066 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 632ba412a681447e93af32d7fdf0691d)
I20250624 07:55:33.671331 15066 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 6 FOLLOWER]: Advancing to term 7
I20250624 07:55:33.673326 15066 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:33.673915 15066 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 election: Requested vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:33.674304 15012 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 7 candidate_status { last_received { term: 6 index: 45 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493"
I20250624 07:55:33.674484 15012 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 6 FOLLOWER]: Advancing to term 7
W20250624 07:55:33.675189 14586 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:33.676578 15012 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588e956d4ac040deb3c06e57bd3c6b88 in term 7.
I20250624 07:55:33.676913 14588 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters: 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:33.677165 15066 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 7 FOLLOWER]: Leader election won for term 7
I20250624 07:55:33.677533 15066 raft_consensus.cc:695] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 7 LEADER]: Becoming Leader. State: Replica: 588e956d4ac040deb3c06e57bd3c6b88, State: Running, Role: LEADER
I20250624 07:55:33.677945 15066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 6.45, Last appended by leader: 45, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:33.679556 15070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588e956d4ac040deb3c06e57bd3c6b88. Latest consensus state: current_term: 7 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:33.679878 15070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:34.097048 15012 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Refusing update from remote peer 588e956d4ac040deb3c06e57bd3c6b88: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 7 index: 46. (index mismatch)
I20250624 07:55:34.097808 15070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 35, Time since last communication: 0.000s
I20250624 07:55:34.107414 15078 mvcc.cc:204] Tried to move back new op lower bound from 7171079087928492032 to 7171079045986635776. Current Snapshot: MvccSnapshot[applied={T|T < 7171079087928492032}]
I20250624 07:55:34.109128 15080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:34.109478 15080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:34.109478 15082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:34.109851 15082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:34.110180 15083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:34.109191 15077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 588e956d4ac040deb3c06e57bd3c6b88. Latest consensus state: current_term: 7 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:34.110523 15083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:34.110541 15077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:34.110940 15077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:34.111089 15077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
W20250624 07:55:34.114405 14586 consensus_peers.cc:487] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 -> Peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Couldn't send request to peer 632ba412a681447e93af32d7fdf0691d. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:55:34.117486 15066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "588e956d4ac040deb3c06e57bd3c6b88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:34.117727 15066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:34.196594 15072 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:34.199235 15072 catalog_manager.cc:671] Loaded metadata for table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:34.199744 15072 catalog_manager.cc:671] Loaded metadata for table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:34.200157 15072 catalog_manager.cc:671] Loaded metadata for table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:34.200623 15072 catalog_manager.cc:671] Loaded metadata for table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:34.201068 15072 catalog_manager.cc:671] Loaded metadata for table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:34.201503 15072 catalog_manager.cc:671] Loaded metadata for table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:34.201885 15072 catalog_manager.cc:671] Loaded metadata for table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:34.202250 15072 catalog_manager.cc:671] Loaded metadata for table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:34.202576 15072 catalog_manager.cc:671] Loaded metadata for table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:34.204417 15072 tablet_loader.cc:96] loaded metadata for tablet 1d7e38715d1543c69f3ceee6fb74d54a (table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939])
I20250624 07:55:34.204924 15072 tablet_loader.cc:96] loaded metadata for tablet 2fc78118d4fa41d0b4a1aabb92616335 (table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e])
I20250624 07:55:34.205220 15072 tablet_loader.cc:96] loaded metadata for tablet 31617090f65045f08d94b793ff525275 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2])
I20250624 07:55:34.205677 15072 tablet_loader.cc:96] loaded metadata for tablet 5daee63b211a4b099533045210d213c8 (table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62])
I20250624 07:55:34.206513 15072 tablet_loader.cc:96] loaded metadata for tablet 8ede62aa66ba4b829be7c283993e3cf7 (table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b])
I20250624 07:55:34.206871 15072 tablet_loader.cc:96] loaded metadata for tablet 901ae0683a114638a6a075cf24c5eabe (table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e])
I20250624 07:55:34.207154 15072 tablet_loader.cc:96] loaded metadata for tablet a7fc1a7fbc17491caa3b44e0a4db6590 (table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d])
I20250624 07:55:34.207720 15072 tablet_loader.cc:96] loaded metadata for tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542])
I20250624 07:55:34.208004 15072 tablet_loader.cc:96] loaded metadata for tablet f5af61fc5261444b8544d72aa61d3082 (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72])
I20250624 07:55:34.208236 15072 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:34.209300 15072 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:34.209386 15072 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:34.210641 15072 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:34.211535 15072 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Loaded TSK: 0
I20250624 07:55:34.213033 15072 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:34.213954 15072 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:34.214793 15072 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250624 07:55:34.295920 14601 master_service.cc:948] ReplaceTablet: received request to replace tablet d377dcea0fc1444abed05b5f8fa6b32c from {username='slave'} at 127.0.0.1:57912
I20250624 07:55:34.296061 14602 master_service.cc:948] ReplaceTablet: received request to replace tablet 75bf5e310d42413ca5da83762bf91e9d from {username='slave'} at 127.0.0.1:57912
I20250624 07:55:34.296947 14598 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57912:
name: "default.Table_17819dd7286645d8a22ec7eed153297a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:34.297464 14597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57912:
name: "default.Table_3e714bb6900c450b90b4519baaaa3aba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:34.297003 14599 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57912:
name: "default.Table_fe124062a0864c5e9472552b1a471845"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:34.299809 14596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_02926d7fe80d4545a80eb082aa7d2671" } modify_external_catalogs: true
I20250624 07:55:34.300045 14596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_02926d7fe80d4545a80eb082aa7d2671" } modify_external_catalogs: true
I20250624 07:55:34.300426 14595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_953a2fe7f1694bb7b1afcbbaddd632b6" } modify_external_catalogs: true
I20250624 07:55:34.300801 14600 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_6e36abec7aef4670b05e620d91bba579" } modify_external_catalogs: true
I20250624 07:55:34.301034 14595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_953a2fe7f1694bb7b1afcbbaddd632b6" } modify_external_catalogs: true
I20250624 07:55:34.301322 14600 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_6e36abec7aef4670b05e620d91bba579" } modify_external_catalogs: true
I20250624 07:55:34.301365 14602 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe" } modify_external_catalogs: true
I20250624 07:55:34.300616 14594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57912:
name: "default.Table_6c5708ec0a8a44f4bc2b8d669d041a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:34.301988 14593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57912:
name: "default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:34.302127 14602 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57912:
table { table_name: "default.Table_b66cb6b1a255496bb4909de6a39e80fe" } modify_external_catalogs: true
F20250624 07:55:34.304677 14656 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f7a833c1dcd google::LogMessage::Fail() at ??:0
@ 0x7f7a833c5b93 google::LogMessage::SendToLog() at ??:0
I20250624 07:55:34.308580 14601 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Sending DeleteTablet for 3 replicas of tablet d377dcea0fc1444abed05b5f8fa6b32c
@ 0x7f7a833c17cc google::LogMessage::Flush() at ??:0
I20250624 07:55:34.311590 14601 catalog_manager.cc:6652] ReplaceTablet: tablet d377dcea0fc1444abed05b5f8fa6b32c deleted and replaced by tablet 819b10f76c2c40c4891d7bd802df5c4d
I20250624 07:55:34.311932 14601 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57912:
name: "default.Table_ec6cab3afe9e48f3a38fa9c0c2ca3922"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:34.328663 13728 tablet_service.cc:1515] Processing DeleteTablet for tablet d377dcea0fc1444abed05b5f8fa6b32c with delete_type TABLET_DATA_DELETED (replaced by tablet 819b10f76c2c40c4891d7bd802df5c4d) from {username='slave'} at 127.0.0.1:50942
I20250624 07:55:34.329046 13590 tablet_service.cc:1515] Processing DeleteTablet for tablet d377dcea0fc1444abed05b5f8fa6b32c with delete_type TABLET_DATA_DELETED (replaced by tablet 819b10f76c2c40c4891d7bd802df5c4d) from {username='slave'} at 127.0.0.1:40842
I20250624 07:55:34.329798 15101 tablet_replica.cc:331] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:34.330041 15101 raft_consensus.cc:2241] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:34.330309 15101 raft_consensus.cc:2270] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:34.330832 15101 ts_tablet_manager.cc:1905] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:34.334713 15102 tablet_replica.cc:331] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: stopping tablet replica
I20250624 07:55:34.334954 15102 raft_consensus.cc:2241] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:34.335393 15102 raft_consensus.cc:2270] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:34.335376 15101 ts_tablet_manager.cc:1918] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:34.335505 15101 log.cc:1199] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/d377dcea0fc1444abed05b5f8fa6b32c
I20250624 07:55:34.335902 15101 ts_tablet_manager.cc:1939] T d377dcea0fc1444abed05b5f8fa6b32c P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:34.336017 15102 ts_tablet_manager.cc:1905] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:34.336266 13863 tablet_service.cc:1515] Processing DeleteTablet for tablet d377dcea0fc1444abed05b5f8fa6b32c with delete_type TABLET_DATA_DELETED (replaced by tablet 819b10f76c2c40c4891d7bd802df5c4d) from {username='slave'} at 127.0.0.1:58532
I20250624 07:55:34.336799 15103 tablet_replica.cc:331] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: stopping tablet replica
I20250624 07:55:34.336967 15103 raft_consensus.cc:2241] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 1 LEADER]: Raft consensus shutting down.
I20250624 07:55:34.337280 15103 raft_consensus.cc:2270] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:34.337720 15103 ts_tablet_manager.cc:1905] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7a833c1a11 google::LogMessage::~LogMessage() at ??:0
I20250624 07:55:34.343020 15103 ts_tablet_manager.cc:1918] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:34.343098 15103 log.cc:1199] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-2/wal/wals/d377dcea0fc1444abed05b5f8fa6b32c
I20250624 07:55:34.343451 15103 ts_tablet_manager.cc:1939] T d377dcea0fc1444abed05b5f8fa6b32c P f96059407ad14a70971e622891d2f99c: Deleting consensus metadata
I20250624 07:55:34.343919 15102 ts_tablet_manager.cc:1918] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:34.344062 15102 log.cc:1199] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-0/wal/wals/d377dcea0fc1444abed05b5f8fa6b32c
I20250624 07:55:34.344440 14587 catalog_manager.cc:4928] TS f96059407ad14a70971e622891d2f99c (127.12.180.3:33179): tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]) successfully deleted
@ 0x7f7a997c26d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250624 07:55:34.345731 15102 ts_tablet_manager.cc:1939] T d377dcea0fc1444abed05b5f8fa6b32c P dbb2615829f748ef929761ed90057cb7: Deleting consensus metadata
I20250624 07:55:34.346590 14586 catalog_manager.cc:4928] TS ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]) successfully deleted
I20250624 07:55:34.346869 14587 catalog_manager.cc:4928] TS dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271): tablet d377dcea0fc1444abed05b5f8fa6b32c (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]) successfully deleted
I20250624 07:55:34.349027 13728 tablet_service.cc:1468] Processing CreateTablet for tablet 2fc78118d4fa41d0b4a1aabb92616335 (DEFAULT_TABLE table=default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.349434 13725 tablet_service.cc:1468] Processing CreateTablet for tablet 1d7e38715d1543c69f3ceee6fb74d54a (DEFAULT_TABLE table=default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.349736 13590 tablet_service.cc:1468] Processing CreateTablet for tablet 2fc78118d4fa41d0b4a1aabb92616335 (DEFAULT_TABLE table=default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.350068 13590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fc78118d4fa41d0b4a1aabb92616335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.350826 13726 tablet_service.cc:1468] Processing CreateTablet for tablet 819b10f76c2c40c4891d7bd802df5c4d (DEFAULT_TABLE table=default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.351408 13592 tablet_service.cc:1468] Processing CreateTablet for tablet 1d7e38715d1543c69f3ceee6fb74d54a (DEFAULT_TABLE table=default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.351429 13727 tablet_service.cc:1468] Processing CreateTablet for tablet a7fc1a7fbc17491caa3b44e0a4db6590 (DEFAULT_TABLE table=default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.351753 13592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7e38715d1543c69f3ceee6fb74d54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.354463 13591 tablet_service.cc:1468] Processing CreateTablet for tablet 819b10f76c2c40c4891d7bd802df5c4d (DEFAULT_TABLE table=default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.354797 13591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819b10f76c2c40c4891d7bd802df5c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.358335 13589 tablet_service.cc:1468] Processing CreateTablet for tablet a7fc1a7fbc17491caa3b44e0a4db6590 (DEFAULT_TABLE table=default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.358913 13589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7fc1a7fbc17491caa3b44e0a4db6590. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7a997c0f81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250624 07:55:34.349431 13728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fc78118d4fa41d0b4a1aabb92616335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.363099 13726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819b10f76c2c40c4891d7bd802df5c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.364913 15106 tablet_bootstrap.cc:492] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:34.366598 13727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7fc1a7fbc17491caa3b44e0a4db6590. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.366859 15106 tablet_bootstrap.cc:654] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.370389 13725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7e38715d1543c69f3ceee6fb74d54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.374413 13864 tablet_service.cc:1468] Processing CreateTablet for tablet 1d7e38715d1543c69f3ceee6fb74d54a (DEFAULT_TABLE table=default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.374818 13864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7e38715d1543c69f3ceee6fb74d54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.378275 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 2fc78118d4fa41d0b4a1aabb92616335 (DEFAULT_TABLE table=default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.378631 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fc78118d4fa41d0b4a1aabb92616335. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f7a997c8bfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7a997c8a79 std::_Function_handler<>::_M_invoke() at ??:0
I20250624 07:55:34.388644 15109 tablet_bootstrap.cc:492] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:34.388576 13865 tablet_service.cc:1468] Processing CreateTablet for tablet a7fc1a7fbc17491caa3b44e0a4db6590 (DEFAULT_TABLE table=default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.388988 13865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7fc1a7fbc17491caa3b44e0a4db6590. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.390643 15109 tablet_bootstrap.cc:654] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.392022 15110 tablet_bootstrap.cc:492] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
@ 0xa7573d std::function<>::operator()() at ??:?
@ 0x7f7a8611b4fe kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7a895396db start_thread at ??:0
I20250624 07:55:34.401589 13863 tablet_service.cc:1468] Processing CreateTablet for tablet 819b10f76c2c40c4891d7bd802df5c4d (DEFAULT_TABLE table=default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 07:55:34.402010 13863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819b10f76c2c40c4891d7bd802df5c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 07:55:34.402616 15110 tablet_bootstrap.cc:654] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.406214 15110 tablet_bootstrap.cc:492] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:34.406344 15110 ts_tablet_manager.cc:1397] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20250624 07:55:34.407152 15110 raft_consensus.cc:357] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.407351 15110 raft_consensus.cc:383] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.407454 15110 raft_consensus.cc:738] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.411018 15110 consensus_queue.cc:260] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.412294 15109 tablet_bootstrap.cc:492] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:34.412449 15109 ts_tablet_manager.cc:1397] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250624 07:55:34.413084 15109 raft_consensus.cc:357] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.413218 15109 raft_consensus.cc:383] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.413293 15109 raft_consensus.cc:738] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.413484 15109 consensus_queue.cc:260] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.413949 15109 ts_tablet_manager.cc:1428] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 07:55:34.414458 13931 heartbeater.cc:499] Master 127.12.180.62:44637 was elected leader, sending a full tablet report...
I20250624 07:55:34.415220 15110 ts_tablet_manager.cc:1428] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250624 07:55:34.415499 15110 tablet_bootstrap.cc:492] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:34.416213 15106 tablet_bootstrap.cc:492] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:34.416355 15106 ts_tablet_manager.cc:1397] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.052s user 0.022s sys 0.026s
I20250624 07:55:34.417109 15106 raft_consensus.cc:357] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.417241 15106 raft_consensus.cc:383] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.417253 15110 tablet_bootstrap.cc:654] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.417296 15106 raft_consensus.cc:738] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.417486 15106 consensus_queue.cc:260] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.418009 15106 ts_tablet_manager.cc:1428] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250624 07:55:34.418115 15109 tablet_bootstrap.cc:492] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:34.418298 15106 tablet_bootstrap.cc:492] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:34.419921 15106 tablet_bootstrap.cc:654] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.420141 15109 tablet_bootstrap.cc:654] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
@ 0x7f7a8227271f clone at ??:0
I20250624 07:55:34.425774 15110 tablet_bootstrap.cc:492] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:34.425956 15110 ts_tablet_manager.cc:1397] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250624 07:55:34.426847 15110 raft_consensus.cc:357] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.427089 15110 raft_consensus.cc:383] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.427191 15110 raft_consensus.cc:738] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.427461 15110 consensus_queue.cc:260] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.428164 15110 ts_tablet_manager.cc:1428] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250624 07:55:34.428452 15110 tablet_bootstrap.cc:492] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:34.429250 15106 tablet_bootstrap.cc:492] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:34.429415 15106 ts_tablet_manager.cc:1397] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20250624 07:55:34.430229 15106 raft_consensus.cc:357] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.430454 15106 raft_consensus.cc:383] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.430536 15106 raft_consensus.cc:738] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.430789 15106 consensus_queue.cc:260] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.431285 15106 ts_tablet_manager.cc:1428] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250624 07:55:34.431625 15106 tablet_bootstrap.cc:492] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:34.433396 15106 tablet_bootstrap.cc:654] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.435585 15109 tablet_bootstrap.cc:492] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:34.435762 15109 ts_tablet_manager.cc:1397] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250624 07:55:34.435876 15106 tablet_bootstrap.cc:492] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:34.436007 15106 ts_tablet_manager.cc:1397] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20250624 07:55:34.436462 15110 tablet_bootstrap.cc:654] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.436502 15109 raft_consensus.cc:357] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.436635 15109 raft_consensus.cc:383] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.436707 15109 raft_consensus.cc:738] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.436714 15106 raft_consensus.cc:357] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.436862 15106 raft_consensus.cc:383] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.436937 15106 raft_consensus.cc:738] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.436914 15109 consensus_queue.cc:260] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.437147 15106 consensus_queue.cc:260] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.437328 15109 ts_tablet_manager.cc:1428] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250624 07:55:34.437589 15109 tablet_bootstrap.cc:492] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:34.437602 15106 ts_tablet_manager.cc:1428] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250624 07:55:34.437840 15106 tablet_bootstrap.cc:492] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7: Bootstrap starting.
I20250624 07:55:34.439522 15106 tablet_bootstrap.cc:654] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.443681 15109 tablet_bootstrap.cc:654] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.445932 15110 tablet_bootstrap.cc:492] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:34.446105 15110 ts_tablet_manager.cc:1397] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20250624 07:55:34.447059 15110 raft_consensus.cc:357] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.447288 15110 raft_consensus.cc:383] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.447386 15110 raft_consensus.cc:738] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.447726 15110 consensus_queue.cc:260] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.448343 15110 ts_tablet_manager.cc:1428] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250624 07:55:34.450768 15110 tablet_bootstrap.cc:492] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c: Bootstrap starting.
I20250624 07:55:34.452193 15109 tablet_bootstrap.cc:492] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:34.452358 15109 ts_tablet_manager.cc:1397] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250624 07:55:34.453132 15109 raft_consensus.cc:357] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.453320 15109 raft_consensus.cc:383] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.453415 15109 raft_consensus.cc:738] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.453645 15110 tablet_bootstrap.cc:654] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.453688 15109 consensus_queue.cc:260] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.459205 15106 tablet_bootstrap.cc:492] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7: No bootstrap required, opened a new log
I20250624 07:55:34.459435 15106 ts_tablet_manager.cc:1397] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20250624 07:55:34.459942 15109 ts_tablet_manager.cc:1428] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250624 07:55:34.460408 15106 raft_consensus.cc:357] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.460666 15106 raft_consensus.cc:383] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.460744 15106 raft_consensus.cc:738] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.461052 15106 consensus_queue.cc:260] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.461899 15106 ts_tablet_manager.cc:1428] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250624 07:55:34.463358 15110 tablet_bootstrap.cc:492] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c: No bootstrap required, opened a new log
I20250624 07:55:34.463506 15110 ts_tablet_manager.cc:1397] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20250624 07:55:34.464196 15110 raft_consensus.cc:357] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.464375 15110 raft_consensus.cc:383] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.464430 15110 raft_consensus.cc:738] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96059407ad14a70971e622891d2f99c, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.464695 15110 consensus_queue.cc:260] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.466370 15110 ts_tablet_manager.cc:1428] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250624 07:55:34.468031 15109 tablet_bootstrap.cc:492] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: Bootstrap starting.
I20250624 07:55:34.469781 15109 tablet_bootstrap.cc:654] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: Neither blocks nor log segments found. Creating new log.
I20250624 07:55:34.471738 15109 tablet_bootstrap.cc:492] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: No bootstrap required, opened a new log
I20250624 07:55:34.471889 15109 ts_tablet_manager.cc:1397] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250624 07:55:34.472734 15109 raft_consensus.cc:357] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.473088 15109 raft_consensus.cc:383] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 07:55:34.473169 15109 raft_consensus.cc:738] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Initialized, Role: FOLLOWER
I20250624 07:55:34.473544 15109 consensus_queue.cc:260] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.476058 15109 ts_tablet_manager.cc:1428] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250624 07:55:34.489171 15076 raft_consensus.cc:491] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:34.489364 15076 raft_consensus.cc:513] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.490197 15076 leader_election.cc:290] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:34.490739 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc78118d4fa41d0b4a1aabb92616335" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:34.490752 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc78118d4fa41d0b4a1aabb92616335" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df" is_pre_election: true
I20250624 07:55:34.490936 13749 raft_consensus.cc:2466] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 0.
I20250624 07:55:34.490947 13885 raft_consensus.cc:2466] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 0.
I20250624 07:55:34.491354 13545 leader_election.cc:304] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:34.491585 15076 raft_consensus.cc:2802] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:34.491694 15076 raft_consensus.cc:491] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:34.491794 15076 raft_consensus.cc:3058] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.493698 15076 raft_consensus.cc:513] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.494259 15076 leader_election.cc:290] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 election: Requested vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:34.494621 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc78118d4fa41d0b4a1aabb92616335" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df"
I20250624 07:55:34.494740 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fc78118d4fa41d0b4a1aabb92616335" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:34.494817 13749 raft_consensus.cc:3058] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.494930 13885 raft_consensus.cc:3058] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.497654 13885 raft_consensus.cc:2466] T 2fc78118d4fa41d0b4a1aabb92616335 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 1.
I20250624 07:55:34.498071 13546 leader_election.cc:304] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbb2615829f748ef929761ed90057cb7, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:34.498236 13749 raft_consensus.cc:2466] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 1.
I20250624 07:55:34.498412 15076 raft_consensus.cc:2802] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:34.498881 15076 raft_consensus.cc:695] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [term 1 LEADER]: Becoming Leader. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Running, Role: LEADER
I20250624 07:55:34.499238 15076 consensus_queue.cc:237] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.504611 15092 raft_consensus.cc:491] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:34.504783 15092 raft_consensus.cc:513] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.505646 15092 leader_election.cc:290] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:34.505990 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7fc1a7fbc17491caa3b44e0a4db6590" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df" is_pre_election: true
I20250624 07:55:34.506229 13749 raft_consensus.cc:2466] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:34.506253 13611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7fc1a7fbc17491caa3b44e0a4db6590" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:34.506501 13611 raft_consensus.cc:2466] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:34.506575 13818 leader_election.cc:304] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:34.506822 15092 raft_consensus.cc:2802] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:34.506929 15092 raft_consensus.cc:491] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:34.507017 15092 raft_consensus.cc:3058] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.508787 15092 raft_consensus.cc:513] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.509268 15092 leader_election.cc:290] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Requested vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:34.509783 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7fc1a7fbc17491caa3b44e0a4db6590" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df"
I20250624 07:55:34.509807 13611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7fc1a7fbc17491caa3b44e0a4db6590" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:34.509971 13749 raft_consensus.cc:3058] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.509986 13611 raft_consensus.cc:3058] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.511776 13749 raft_consensus.cc:2466] T a7fc1a7fbc17491caa3b44e0a4db6590 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
I20250624 07:55:34.512087 13818 leader_election.cc:304] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:34.512362 15092 raft_consensus.cc:2802] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:34.512800 15092 raft_consensus.cc:695] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [term 1 LEADER]: Becoming Leader. State: Replica: f96059407ad14a70971e622891d2f99c, State: Running, Role: LEADER
I20250624 07:55:34.512882 13611 raft_consensus.cc:2466] T a7fc1a7fbc17491caa3b44e0a4db6590 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
I20250624 07:55:34.513108 15092 consensus_queue.cc:237] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.591266 15092 raft_consensus.cc:491] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:34.591493 15092 raft_consensus.cc:513] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.592182 15092 leader_election.cc:290] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:34.592742 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819b10f76c2c40c4891d7bd802df5c4d" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df" is_pre_election: true
I20250624 07:55:34.592831 13611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819b10f76c2c40c4891d7bd802df5c4d" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:34.592981 13749 raft_consensus.cc:2466] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:34.593034 13611 raft_consensus.cc:2466] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 0.
I20250624 07:55:34.593403 13818 leader_election.cc:304] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:34.593714 15092 raft_consensus.cc:2802] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:34.593804 15092 raft_consensus.cc:491] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:34.593896 15092 raft_consensus.cc:3058] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.595791 15092 raft_consensus.cc:513] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.596357 15092 leader_election.cc:290] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Requested vote from peers ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137), dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:34.596765 13749 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819b10f76c2c40c4891d7bd802df5c4d" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab43ace4a8b44b908d247cad235e06df"
I20250624 07:55:34.596818 13611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819b10f76c2c40c4891d7bd802df5c4d" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:34.596952 13749 raft_consensus.cc:3058] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.597007 13611 raft_consensus.cc:3058] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.599735 13749 raft_consensus.cc:2466] T 819b10f76c2c40c4891d7bd802df5c4d P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
I20250624 07:55:34.600162 13818 leader_election.cc:304] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:34.600225 13611 raft_consensus.cc:2466] T 819b10f76c2c40c4891d7bd802df5c4d P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f96059407ad14a70971e622891d2f99c in term 1.
I20250624 07:55:34.600427 15092 raft_consensus.cc:2802] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:34.600813 15092 raft_consensus.cc:695] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [term 1 LEADER]: Becoming Leader. State: Replica: f96059407ad14a70971e622891d2f99c, State: Running, Role: LEADER
I20250624 07:55:34.601187 15092 consensus_queue.cc:237] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
W20250624 07:55:34.622339 13940 connection.cc:537] client connection to 127.12.180.62:44637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:34.622651 15091 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:34.622040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.180.62:44637 (local address 127.12.180.3:40417)
0624 07:55:34.622267 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0624 07:55:34.622414 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":108}
W20250624 07:55:34.622653 15089 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:34.622040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.180.62:44637 (local address 127.12.180.1:59101)
0624 07:55:34.622286 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0624 07:55:34.622427 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":107}
W20250624 07:55:34.623041 13940 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.180.62:44637)
W20250624 07:55:34.623679 13940 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.180.62:44637)
W20250624 07:55:34.623898 13940 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.180.62:44637)
W20250624 07:55:34.624565 13940 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.180.62:44637)
I20250624 07:55:34.781463 14965 raft_consensus.cc:491] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:34.781678 14965 raft_consensus.cc:513] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.782467 14965 leader_election.cc:290] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:34.782985 13611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7e38715d1543c69f3ceee6fb74d54a" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
I20250624 07:55:34.783022 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7e38715d1543c69f3ceee6fb74d54a" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:34.783164 13611 raft_consensus.cc:2466] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:34.783202 13885 raft_consensus.cc:2466] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 0.
I20250624 07:55:34.783589 13683 leader_election.cc:304] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, f96059407ad14a70971e622891d2f99c; no voters:
I20250624 07:55:34.783834 14965 raft_consensus.cc:2802] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 07:55:34.783946 14965 raft_consensus.cc:491] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:34.784037 14965 raft_consensus.cc:3058] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.785805 14965 raft_consensus.cc:513] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.786298 14965 leader_election.cc:290] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Requested vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:34.786671 13611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7e38715d1543c69f3ceee6fb74d54a" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7"
I20250624 07:55:34.786738 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d7e38715d1543c69f3ceee6fb74d54a" candidate_uuid: "ab43ace4a8b44b908d247cad235e06df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:34.786818 13611 raft_consensus.cc:3058] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.786936 13885 raft_consensus.cc:3058] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c [term 0 FOLLOWER]: Advancing to term 1
I20250624 07:55:34.788618 13611 raft_consensus.cc:2466] T 1d7e38715d1543c69f3ceee6fb74d54a P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:34.788838 13885 raft_consensus.cc:2466] T 1d7e38715d1543c69f3ceee6fb74d54a P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab43ace4a8b44b908d247cad235e06df in term 1.
I20250624 07:55:34.789006 13683 leader_election.cc:304] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7; no voters:
I20250624 07:55:34.789263 14965 raft_consensus.cc:2802] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Leader election won for term 1
I20250624 07:55:34.789640 14965 raft_consensus.cc:695] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Becoming Leader. State: Replica: ab43ace4a8b44b908d247cad235e06df, State: Running, Role: LEADER
I20250624 07:55:34.789949 14965 consensus_queue.cc:237] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } }
I20250624 07:55:34.848098 15136 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 588e956d4ac040deb3c06e57bd3c6b88)
I20250624 07:55:34.848289 15136 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:34.848902 15136 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:34.850828 14989 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:34.850999 14986 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:34.851099 14986 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493; no voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:34.851395 15136 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250624 07:55:35.432327 15140 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 588e956d4ac040deb3c06e57bd3c6b88)
I20250624 07:55:35.432641 15140 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:35.433238 15140 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:35.434864 14989 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637): Network error: Client connection negotiation failed: client connection to 127.12.180.62:44637: connect: Connection refused (error 111)
W20250624 07:55:35.435030 14986 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:35.435142 14986 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493; no voters: 588e956d4ac040deb3c06e57bd3c6b88, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:35.435388 15140 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250624 07:55:35.519579 15144 consensus_queue.cc:1035] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:35.526577 15144 consensus_queue.cc:1035] T a7fc1a7fbc17491caa3b44e0a4db6590 P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:35.527634 15076 consensus_queue.cc:1035] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:35.528113 15076 consensus_queue.cc:1035] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:35.548888 15144 consensus_queue.cc:1035] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:35.553755 15144 consensus_queue.cc:1035] T 819b10f76c2c40c4891d7bd802df5c4d P f96059407ad14a70971e622891d2f99c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:35.587888 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--webserver_interface=127.12.180.62
--webserver_port=45479
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250624 07:55:35.641101 14965 consensus_queue.cc:1035] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 07:55:35.650117 15165 consensus_queue.cc:1035] T 1d7e38715d1543c69f3ceee6fb74d54a P ab43ace4a8b44b908d247cad235e06df [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250624 07:55:35.912643 15163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:35.913089 15163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:35.913200 15163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:35.923723 15163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:35.923821 15163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:35.923890 15163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:35.923928 15163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:35.933456 15163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.62:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.180.62
--webserver_port=45479
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:35.935354 15163 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:35.937057 15163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:35.946803 15174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:35.946789 15173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:35.946810 15176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:35.947852 15163 server_base.cc:1048] running on GCE node
I20250624 07:55:35.948697 15163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:35.950491 15163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:35.951678 15163 hybrid_clock.cc:648] HybridClock initialized: now 1750751735951636 us; error 29 us; skew 500 ppm
I20250624 07:55:35.952018 15163 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:35.954324 15163 webserver.cc:469] Webserver started at http://127.12.180.62:45479/ using document root <none> and password file <none>
I20250624 07:55:35.955073 15163 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:35.955190 15163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:35.960291 15163 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250624 07:55:35.962978 15182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:35.964166 15163 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250624 07:55:35.964318 15163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
uuid: "588e956d4ac040deb3c06e57bd3c6b88"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:35.964932 15163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:35.995452 15163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:35.996369 15163 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:35.996630 15163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:36.015455 15163 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.62:44637
I20250624 07:55:36.015476 15233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.62:44637 every 8 connection(s)
I20250624 07:55:36.017076 15163 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-0/data/info.pb
I20250624 07:55:36.023664 15234 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:36.026866 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 15163
I20250624 07:55:36.027374 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap starting.
I20250624 07:55:36.043151 15234 log.cc:826] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:36.078588 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap replayed 1/1 log segments. Stats: ops{read=48 overwritten=0 applied=48 ignored=0} inserts{seen=30 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250624 07:55:36.079085 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: Bootstrap complete.
I20250624 07:55:36.084065 15234 raft_consensus.cc:357] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:36.084699 15234 raft_consensus.cc:738] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588e956d4ac040deb3c06e57bd3c6b88, State: Initialized, Role: FOLLOWER
I20250624 07:55:36.085282 15234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 48, Last appended: 7.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:36.085954 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:36.086279 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:36.086972 15234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:36.099015 15252 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:36.099192 15252 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: loading cluster ID for follower catalog manager: success
I20250624 07:55:36.103242 15252 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: acquiring CA information for follower catalog manager: success
I20250624 07:55:36.107439 15252 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:36.127054 15163 master_runner.cc:186] Error getting master registration for 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111),
I20250624 07:55:36.130228 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:36.157506 13931 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:36.157730 13931 heartbeater.cc:499] Master 127.12.180.62:44637 was elected leader, sending a full tablet report...
I20250624 07:55:36.158643 13795 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:36.161381 15198 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:43989; Asking this server to re-register.
I20250624 07:55:36.161399 15199 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:47933; Asking this server to re-register.
I20250624 07:55:36.162346 13795 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:36.162513 13931 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:36.164619 15198 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:36.164680 15199 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:36.171823 13658 heartbeater.cc:344] Connected to a master server at 127.12.180.62:44637
I20250624 07:55:36.173580 15199 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:37767; Asking this server to re-register.
I20250624 07:55:36.174155 13658 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:36.175511 15199 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:36.283707 15261 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 588e956d4ac040deb3c06e57bd3c6b88)
I20250624 07:55:36.283918 15261 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:36.284696 15261 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:36.287411 14986 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:36.292013 15209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 8 candidate_status { last_received { term: 7 index: 48 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
I20250624 07:55:36.292666 15209 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52afd679ffb241a3a9022cc850c03493 in term 7.
I20250624 07:55:36.293258 14989 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters: 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:36.293578 15261 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250624 07:55:36.293702 15261 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 588e956d4ac040deb3c06e57bd3c6b88)
I20250624 07:55:36.293800 15261 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 7 FOLLOWER]: Advancing to term 8
I20250624 07:55:36.296901 15261 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:36.297699 15261 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 election: Requested vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:36.298235 15209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 8 candidate_status { last_received { term: 7 index: 48 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88"
I20250624 07:55:36.298518 15209 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 7 FOLLOWER]: Advancing to term 8
W20250624 07:55:36.299167 14986 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:36.304255 15209 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52afd679ffb241a3a9022cc850c03493 in term 8.
I20250624 07:55:36.304740 14989 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters: 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:36.305047 15261 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Leader election won for term 8
I20250624 07:55:36.305487 15261 raft_consensus.cc:695] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 LEADER]: Becoming Leader. State: Replica: 52afd679ffb241a3a9022cc850c03493, State: Running, Role: LEADER
I20250624 07:55:36.306006 15261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 7.48, Last appended by leader: 48, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:36.307973 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52afd679ffb241a3a9022cc850c03493. Latest consensus state: current_term: 8 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:36.308271 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:36.308900 15267 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:36.311322 15267 catalog_manager.cc:671] Loaded metadata for table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:36.311722 15267 catalog_manager.cc:671] Loaded metadata for table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:36.312062 15267 catalog_manager.cc:671] Loaded metadata for table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:36.312423 15267 catalog_manager.cc:671] Loaded metadata for table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:36.312731 15267 catalog_manager.cc:671] Loaded metadata for table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:36.313073 15267 catalog_manager.cc:671] Loaded metadata for table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:36.313398 15267 catalog_manager.cc:671] Loaded metadata for table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:36.313705 15267 catalog_manager.cc:671] Loaded metadata for table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:36.314050 15267 catalog_manager.cc:671] Loaded metadata for table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:36.315995 15267 tablet_loader.cc:96] loaded metadata for tablet 1d7e38715d1543c69f3ceee6fb74d54a (table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939])
I20250624 07:55:36.316490 15267 tablet_loader.cc:96] loaded metadata for tablet 2fc78118d4fa41d0b4a1aabb92616335 (table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e])
I20250624 07:55:36.316762 15267 tablet_loader.cc:96] loaded metadata for tablet 31617090f65045f08d94b793ff525275 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2])
I20250624 07:55:36.317133 15267 tablet_loader.cc:96] loaded metadata for tablet 5daee63b211a4b099533045210d213c8 (table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62])
I20250624 07:55:36.317663 15267 tablet_loader.cc:96] loaded metadata for tablet 819b10f76c2c40c4891d7bd802df5c4d (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542])
I20250624 07:55:36.318096 15267 tablet_loader.cc:96] loaded metadata for tablet 8ede62aa66ba4b829be7c283993e3cf7 (table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b])
I20250624 07:55:36.318400 15267 tablet_loader.cc:96] loaded metadata for tablet 901ae0683a114638a6a075cf24c5eabe (table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e])
I20250624 07:55:36.318650 15267 tablet_loader.cc:96] loaded metadata for tablet a7fc1a7fbc17491caa3b44e0a4db6590 (table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d])
I20250624 07:55:36.319337 15267 tablet_loader.cc:96] loaded metadata for tablet f5af61fc5261444b8544d72aa61d3082 (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72])
I20250624 07:55:36.319558 15267 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:36.320541 15267 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:36.320609 15267 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:36.321797 15267 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:36.322670 15267 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Loaded TSK: 0
I20250624 07:55:36.323423 15267 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:36.324321 15267 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:36.325181 15267 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
I20250624 07:55:36.346611 15001 master_service.cc:948] ReplaceTablet: received request to replace tablet 2fc78118d4fa41d0b4a1aabb92616335 from {username='slave'} at 127.0.0.1:35748
I20250624 07:55:36.346977 14999 master_service.cc:948] ReplaceTablet: received request to replace tablet 75bf5e310d42413ca5da83762bf91e9d from {username='slave'} at 127.0.0.1:35748
I20250624 07:55:36.348049 14997 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35748:
table { table_name: "default.Table_8a84bf94dbd94e65946d5e56b69172be" } new_table_name: "default.Table_d133c18e2676484dabe455c76b424d05" modify_external_catalogs: true
I20250624 07:55:36.348497 15002 master_service.cc:948] ReplaceTablet: received request to replace tablet 5daee63b211a4b099533045210d213c8 from {username='slave'} at 127.0.0.1:35748
I20250624 07:55:36.348749 14995 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35748:
table { table_name: "default.Table_2974357596ca4893b46d7199b1367e2f" } new_table_name: "default.Table_77a9deb86184406588f30ae47cdb52d6" modify_external_catalogs: true
I20250624 07:55:36.349378 15000 master_service.cc:948] ReplaceTablet: received request to replace tablet 5daee63b211a4b099533045210d213c8 from {username='slave'} at 127.0.0.1:35748
I20250624 07:55:36.349700 14998 master_service.cc:948] ReplaceTablet: received request to replace tablet 5daee63b211a4b099533045210d213c8 from {username='slave'} at 127.0.0.1:35748
I20250624 07:55:36.359745 14993 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35748:
table { table_name: "default.Table_0047240c5fdf4b85bc2b5fda6bdcef32" } new_table_name: "default.Table_34f740d34f55471b82a8fd8be1423bbf" modify_external_catalogs: true
I20250624 07:55:36.360435 14999 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35748:
name: "default.Table_3e714bb6900c450b90b4519baaaa3aba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:36.361006 14996 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35748:
name: "default.Table_258e5344814340e49754e36a4631c5e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:36.361797 14994 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35748:
name: "default.Table_26071aa0d9c8459b8f1d9583dedee21a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:36.366688 15209 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Refusing update from remote peer 52afd679ffb241a3a9022cc850c03493: Log matching property violated. Preceding OpId in replica: term: 7 index: 48. Preceding OpId from leader: term: 8 index: 50. (index mismatch)
I20250624 07:55:36.367630 15265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
F20250624 07:55:36.375322 15055 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7ff25f81adcd google::LogMessage::Fail() at ??:0
W20250624 07:55:36.381399 14986 consensus_peers.cc:487] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 -> Peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Couldn't send request to peer 632ba412a681447e93af32d7fdf0691d. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7ff25f81eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff25f81a7cc google::LogMessage::Flush() at ??:0
@ 0x7ff25f81aa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ff275c1b6d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250624 07:55:36.419430 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52afd679ffb241a3a9022cc850c03493. Latest consensus state: current_term: 8 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:36.419705 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7ff275c19f81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ff275c21bfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250624 07:55:36.432564 15001 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Sending DeleteTablet for 3 replicas of tablet 2fc78118d4fa41d0b4a1aabb92616335
I20250624 07:55:36.432830 15002 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Sending DeleteTablet for 3 replicas of tablet 5daee63b211a4b099533045210d213c8
@ 0x7ff275c21a79 std::_Function_handler<>::_M_invoke() at ??:0
I20250624 07:55:36.434731 15001 catalog_manager.cc:6652] ReplaceTablet: tablet 2fc78118d4fa41d0b4a1aabb92616335 deleted and replaced by tablet 66cf2fd1ecd74a2b9d9e098cea718c7d
@ 0xa7573d std::function<>::operator()() at ??:?
I20250624 07:55:36.440343 15002 catalog_manager.cc:6652] ReplaceTablet: tablet 5daee63b211a4b099533045210d213c8 deleted and replaced by tablet 601669524f5447929fa9a18597f31d61
I20250624 07:55:36.445803 15261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:36.446072 15261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:36.446305 15000 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35748:
name: "default.Table_17819dd7286645d8a22ec7eed153297a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:36.448632 15001 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35748:
name: "default.Table_fe124062a0864c5e9472552b1a471845"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250624 07:55:36.465409 13726 tablet_service.cc:1515] Processing DeleteTablet for tablet 5daee63b211a4b099533045210d213c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 601669524f5447929fa9a18597f31d61) from {username='slave'} at 127.0.0.1:50944
I20250624 07:55:36.465444 13728 tablet_service.cc:1515] Processing DeleteTablet for tablet 2fc78118d4fa41d0b4a1aabb92616335 with delete_type TABLET_DATA_DELETED (replaced by tablet 66cf2fd1ecd74a2b9d9e098cea718c7d) from {username='slave'} at 127.0.0.1:50944
I20250624 07:55:36.466696 15002 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35748:
table { table_name: "default.Table_6e36abec7aef4670b05e620d91bba579" } modify_external_catalogs: true
I20250624 07:55:36.467023 15002 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35748:
table { table_name: "default.Table_6e36abec7aef4670b05e620d91bba579" } modify_external_catalogs: true
I20250624 07:55:36.469518 14998 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35748:
table { table_name: "default.Table_02926d7fe80d4545a80eb082aa7d2671" } modify_external_catalogs: true
I20250624 07:55:36.469781 15283 tablet_replica.cc:331] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:36.469802 14998 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35748:
table { table_name: "default.Table_02926d7fe80d4545a80eb082aa7d2671" } modify_external_catalogs: true
I20250624 07:55:36.469986 15283 raft_consensus.cc:2241] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 1 LEADER]: Raft consensus shutting down.
@ 0x7ff2625744fe kudu::Thread::SuperviseThread() at ??:0
I20250624 07:55:36.470414 15283 raft_consensus.cc:2270] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:36.471006 15283 ts_tablet_manager.cc:1905] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7ff2659926db start_thread at ??:0
@ 0x7ff25e6cb71f clone at ??:0
I20250624 07:55:36.479470 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:36.479696 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:36.482015 15283 ts_tablet_manager.cc:1918] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:36.482141 15283 log.cc:1199] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/5daee63b211a4b099533045210d213c8
I20250624 07:55:36.482592 15283 ts_tablet_manager.cc:1939] T 5daee63b211a4b099533045210d213c8 P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
I20250624 07:55:36.483717 15283 tablet_replica.cc:331] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: stopping tablet replica
I20250624 07:55:36.483953 15283 raft_consensus.cc:2241] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus shutting down.
I20250624 07:55:36.484144 15283 raft_consensus.cc:2270] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 07:55:36.485000 15283 ts_tablet_manager.cc:1905] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250624 07:55:36.493716 15283 ts_tablet_manager.cc:1918] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250624 07:55:36.493840 15283 log.cc:1199] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: Deleting WAL directory at /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/ts-1/wal/wals/2fc78118d4fa41d0b4a1aabb92616335
I20250624 07:55:36.494258 15283 ts_tablet_manager.cc:1939] T 2fc78118d4fa41d0b4a1aabb92616335 P ab43ace4a8b44b908d247cad235e06df: Deleting consensus metadata
W20250624 07:55:36.665176 13682 connection.cc:537] server connection from 127.0.0.1:50944 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:36.665427 15185 connection.cc:537] server connection from 127.0.0.1:48878 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:36.665493 15279 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:36.436867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.12.180.1:39271)
0624 07:55:36.462908 (+ 26041us) server_negotiation.cc:206] Beginning negotiation
0624 07:55:36.462915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0624 07:55:36.462938 (+ 23us) server_negotiation.cc:407] Connection header received
0624 07:55:36.463007 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0624 07:55:36.463012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0624 07:55:36.463095 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0624 07:55:36.474665 (+ 11570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0624 07:55:36.479169 (+ 4504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0624 07:55:36.480923 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0624 07:55:36.665200 (+184277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40850: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":25925,"thread_start_us":26870,"threads_started":1}
W20250624 07:55:36.665573 15278 negotiation.cc:337] Failed RPC negotiation. Trace:
0624 07:55:36.436622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.12.180.3:33179)
0624 07:55:36.464132 (+ 27510us) server_negotiation.cc:206] Beginning negotiation
0624 07:55:36.464138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0624 07:55:36.464157 (+ 19us) server_negotiation.cc:407] Connection header received
0624 07:55:36.464222 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0624 07:55:36.464227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0624 07:55:36.464299 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0624 07:55:36.474974 (+ 10675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0624 07:55:36.481174 (+ 6200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0624 07:55:36.482709 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0624 07:55:36.665249 (+182540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58536: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":27390,"thread_start_us":37823,"threads_started":1}
W20250624 07:55:36.665784 13820 connection.cc:537] client connection to 127.12.180.61:46565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:36.666149 13684 connection.cc:537] client connection to 127.12.180.61:46565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:36.666906 13939 connection.cc:537] client connection to 127.12.180.61:46565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:36.667582 13939 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.12.180.61:46565)
W20250624 07:55:36.668207 13939 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.12.180.61:46565)
W20250624 07:55:36.668466 13939 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.180.61:46565)
W20250624 07:55:36.668782 13939 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.12.180.61:46565)
I20250624 07:55:36.922554 15238 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:36.922770 15238 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:36.924530 15238 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:36.925371 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:36.926582 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:36.926764 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:36.926885 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:36.927201 15238 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250624 07:55:37.152518 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 1: this message will repeat every 5th retry.
I20250624 07:55:37.507227 15303 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:37.507387 15303 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:37.507937 15303 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:37.510083 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:37.510282 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:37.510408 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:37.510638 15303 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250624 07:55:38.513626 15314 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:38.513803 15314 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:38.514441 15314 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:38.516404 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:38.516561 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:38.516690 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:38.516878 15314 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250624 07:55:38.798264 15302 raft_consensus.cc:491] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab43ace4a8b44b908d247cad235e06df)
I20250624 07:55:38.798467 15302 raft_consensus.cc:513] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:38.799059 15302 leader_election.cc:290] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:38.799516 13611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5daee63b211a4b099533045210d213c8" candidate_uuid: "f96059407ad14a70971e622891d2f99c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dbb2615829f748ef929761ed90057cb7" is_pre_election: true
W20250624 07:55:38.799933 13818 leader_election.cc:343] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Not found: Tablet not found: 5daee63b211a4b099533045210d213c8
I20250624 07:55:38.800098 13819 leader_election.cc:304] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f96059407ad14a70971e622891d2f99c; no voters: ab43ace4a8b44b908d247cad235e06df, dbb2615829f748ef929761ed90057cb7
I20250624 07:55:38.800343 15302 raft_consensus.cc:2747] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250624 07:55:38.813050 15307 raft_consensus.cc:491] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab43ace4a8b44b908d247cad235e06df)
I20250624 07:55:38.813272 15307 raft_consensus.cc:513] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:38.813891 15307 leader_election.cc:290] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f96059407ad14a70971e622891d2f99c (127.12.180.3:33179), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:38.814460 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5daee63b211a4b099533045210d213c8" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c" is_pre_election: true
I20250624 07:55:38.814678 13885 raft_consensus.cc:2466] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 1.
W20250624 07:55:38.814788 13545 leader_election.cc:343] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Not found: Tablet not found: 5daee63b211a4b099533045210d213c8
I20250624 07:55:38.815138 13546 leader_election.cc:304] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dbb2615829f748ef929761ed90057cb7, f96059407ad14a70971e622891d2f99c; no voters: ab43ace4a8b44b908d247cad235e06df
I20250624 07:55:38.815369 15307 raft_consensus.cc:2802] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250624 07:55:38.815461 15307 raft_consensus.cc:491] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab43ace4a8b44b908d247cad235e06df)
I20250624 07:55:38.815557 15307 raft_consensus.cc:3058] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 1 FOLLOWER]: Advancing to term 2
I20250624 07:55:38.817344 15307 raft_consensus.cc:513] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:38.817898 15307 leader_election.cc:290] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 2 election: Requested vote from peers f96059407ad14a70971e622891d2f99c (127.12.180.3:33179), ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:38.818399 13885 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5daee63b211a4b099533045210d213c8" candidate_uuid: "dbb2615829f748ef929761ed90057cb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f96059407ad14a70971e622891d2f99c"
I20250624 07:55:38.818589 13885 raft_consensus.cc:3058] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 1 FOLLOWER]: Advancing to term 2
W20250624 07:55:38.818804 13545 leader_election.cc:343] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Not found: Tablet not found: 5daee63b211a4b099533045210d213c8
I20250624 07:55:38.820338 13885 raft_consensus.cc:2466] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb2615829f748ef929761ed90057cb7 in term 2.
I20250624 07:55:38.820674 13546 leader_election.cc:304] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dbb2615829f748ef929761ed90057cb7, f96059407ad14a70971e622891d2f99c; no voters: ab43ace4a8b44b908d247cad235e06df
I20250624 07:55:38.820935 15307 raft_consensus.cc:2802] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 2 FOLLOWER]: Leader election won for term 2
I20250624 07:55:38.821331 15307 raft_consensus.cc:695] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [term 2 LEADER]: Becoming Leader. State: Replica: dbb2615829f748ef929761ed90057cb7, State: Running, Role: LEADER
I20250624 07:55:38.821914 15307 consensus_queue.cc:237] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 } } peers { permanent_uuid: "dbb2615829f748ef929761ed90057cb7" member_type: VOTER last_known_addr { host: "127.12.180.1" port: 39271 } } peers { permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" member_type: VOTER last_known_addr { host: "127.12.180.2" port: 43137 } }
I20250624 07:55:39.306296 13008 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
/tmp/dist-test-taskPG5ypB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.61:46565
--webserver_interface=127.12.180.61
--webserver_port=43475
--builtin_ntp_servers=127.12.180.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250624 07:55:39.615830 15322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 07:55:39.616201 15322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 07:55:39.616313 15322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 07:55:39.623492 15322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 07:55:39.623625 15322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 07:55:39.623728 15322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 07:55:39.623792 15322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 07:55:39.633431 15322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.180.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37621
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.180.61:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.180.61
--webserver_port=43475
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 07:43:20 UTC on 24a791456cd2
build id 6744
ASAN enabled
I20250624 07:55:39.635092 15322 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 07:55:39.637230 15322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 07:55:39.646987 15330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:39.647248 15329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 07:55:39.647284 15332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 07:55:39.648445 15322 server_base.cc:1048] running on GCE node
I20250624 07:55:39.649343 15322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 07:55:39.651366 15322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 07:55:39.652595 15322 hybrid_clock.cc:648] HybridClock initialized: now 1750751739652513 us; error 80 us; skew 500 ppm
I20250624 07:55:39.652961 15322 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 07:55:39.659281 15322 webserver.cc:469] Webserver started at http://127.12.180.61:43475/ using document root <none> and password file <none>
I20250624 07:55:39.660082 15322 fs_manager.cc:362] Metadata directory not provided
I20250624 07:55:39.660204 15322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 07:55:39.665272 15322 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20250624 07:55:39.668105 15338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 07:55:39.669313 15322 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250624 07:55:39.669461 15322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data,/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
uuid: "52afd679ffb241a3a9022cc850c03493"
format_stamp: "Formatted at 2025-06-24 07:55:07 on dist-test-slave-68nn"
I20250624 07:55:39.670075 15322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 07:55:39.689100 15322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 07:55:39.689953 15322 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 07:55:39.690261 15322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 07:55:39.709245 15322 rpc_server.cc:307] RPC server started. Bound to: 127.12.180.61:46565
I20250624 07:55:39.709324 15389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.180.61:46565 every 8 connection(s)
I20250624 07:55:39.710929 15322 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/master-1/data/info.pb
I20250624 07:55:39.715361 13008 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskPG5ypB/build/asan/bin/kudu as pid 15322
I20250624 07:55:39.717387 15390 sys_catalog.cc:263] Verifying existing consensus state
I20250624 07:55:39.720854 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Bootstrap starting.
I20250624 07:55:39.736595 15390 log.cc:826] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Log is configured to *not* fsync() on all Append() calls
I20250624 07:55:39.748025 15394 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:39.748220 15394 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:39.748804 15394 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:39.758991 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:39.781817 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=51 ignored=0} inserts{seen=32 ignored=0} mutations{seen=36 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250624 07:55:39.782619 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Bootstrap complete.
I20250624 07:55:39.790035 15390 raft_consensus.cc:357] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:39.793413 15390 raft_consensus.cc:738] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52afd679ffb241a3a9022cc850c03493, State: Initialized, Role: FOLLOWER
I20250624 07:55:39.794086 15390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 8.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:39.794914 15399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:39.795228 15399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:39.796039 15390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 07:55:39.796156 15365 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "588e956d4ac040deb3c06e57bd3c6b88" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "52afd679ffb241a3a9022cc850c03493" is_pre_election: true
I20250624 07:55:39.796653 15365 raft_consensus.cc:2408] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 588e956d4ac040deb3c06e57bd3c6b88 for term 9 because replica has last-logged OpId of term: 8 index: 52, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 51.
I20250624 07:55:39.797384 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:39.797780 15394 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250624 07:55:39.822153 15413 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:39.822295 15413 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: loading cluster ID for follower catalog manager: success
I20250624 07:55:39.826480 15413 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: acquiring CA information for follower catalog manager: success
I20250624 07:55:39.828835 15413 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250624 07:55:39.837019 15322 master_runner.cc:186] Error getting master registration for 127.12.180.60:33449: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111),
I20250624 07:55:39.840466 15322 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:39.932826 13659 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:39.935670 15355 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dbb2615829f748ef929761ed90057cb7" instance_seqno: 1750751708877360) as {username='slave'} at 127.12.180.1:51765; Asking this server to re-register.
I20250624 07:55:39.936358 13659 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:39.937846 15355 ts_manager.cc:194] Registered new tserver with Master: dbb2615829f748ef929761ed90057cb7 (127.12.180.1:39271)
I20250624 07:55:39.965790 13885 raft_consensus.cc:1273] T 5daee63b211a4b099533045210d213c8 P f96059407ad14a70971e622891d2f99c [term 2 FOLLOWER]: Refusing update from remote peer dbb2615829f748ef929761ed90057cb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250624 07:55:39.966498 15318 consensus_queue.cc:1035] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96059407ad14a70971e622891d2f99c" member_type: VOTER last_known_addr { host: "127.12.180.3" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250624 07:55:39.976295 13932 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:39.978778 15355 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f96059407ad14a70971e622891d2f99c" instance_seqno: 1750751709829722) as {username='slave'} at 127.12.180.3:56427; Asking this server to re-register.
I20250624 07:55:39.979604 13932 heartbeater.cc:461] Registering TS with master...
W20250624 07:55:39.979676 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 1: this message will repeat every 5th retry.
I20250624 07:55:39.982016 15353 ts_manager.cc:194] Registered new tserver with Master: f96059407ad14a70971e622891d2f99c (127.12.180.3:33179)
I20250624 07:55:40.181936 13796 heartbeater.cc:344] Connected to a master server at 127.12.180.61:46565
I20250624 07:55:40.183503 15355 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab43ace4a8b44b908d247cad235e06df" instance_seqno: 1750751709360311) as {username='slave'} at 127.12.180.2:36069; Asking this server to re-register.
I20250624 07:55:40.184211 13796 heartbeater.cc:461] Registering TS with master...
I20250624 07:55:40.185606 15355 ts_manager.cc:194] Registered new tserver with Master: ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137)
I20250624 07:55:40.223692 15399 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 07:55:40.224025 15399 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:40.226030 15399 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:40.226459 15208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 9 candidate_status { last_received { term: 8 index: 52 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88" is_pre_election: true
I20250624 07:55:40.226753 15208 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52afd679ffb241a3a9022cc850c03493 in term 8.
W20250624 07:55:40.226929 15339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:40.227376 15342 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters:
I20250624 07:55:40.227834 15399 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250624 07:55:40.227972 15399 raft_consensus.cc:491] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 07:55:40.228107 15399 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 8 FOLLOWER]: Advancing to term 9
W20250624 07:55:40.228561 15339 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:40.232599 15399 raft_consensus.cc:513] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:40.233278 15399 leader_election.cc:290] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 9 election: Requested vote from peers 588e956d4ac040deb3c06e57bd3c6b88 (127.12.180.62:44637), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
I20250624 07:55:40.233651 15208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "52afd679ffb241a3a9022cc850c03493" candidate_term: 9 candidate_status { last_received { term: 8 index: 52 } } ignore_live_leader: false dest_uuid: "588e956d4ac040deb3c06e57bd3c6b88"
I20250624 07:55:40.233834 15208 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 8 FOLLOWER]: Advancing to term 9
W20250624 07:55:40.234584 15339 leader_election.cc:336] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:40.235976 15208 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52afd679ffb241a3a9022cc850c03493 in term 9.
I20250624 07:55:40.236373 15342 leader_election.cc:304] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52afd679ffb241a3a9022cc850c03493, 588e956d4ac040deb3c06e57bd3c6b88; no voters: 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:40.236668 15399 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 9 FOLLOWER]: Leader election won for term 9
I20250624 07:55:40.237068 15399 raft_consensus.cc:695] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [term 9 LEADER]: Becoming Leader. State: Replica: 52afd679ffb241a3a9022cc850c03493, State: Running, Role: LEADER
I20250624 07:55:40.237475 15399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 8.52, Last appended by leader: 52, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:40.239444 15430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52afd679ffb241a3a9022cc850c03493. Latest consensus state: current_term: 9 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:40.239768 15430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: LEADER
W20250624 07:55:40.724567 15339 consensus_peers.cc:487] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 -> Peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Couldn't send request to peer 632ba412a681447e93af32d7fdf0691d. Status: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250624 07:55:40.731037 15208 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Refusing update from remote peer 52afd679ffb241a3a9022cc850c03493: Log matching property violated. Preceding OpId in replica: term: 8 index: 51. Preceding OpId from leader: term: 9 index: 53. (index mismatch)
I20250624 07:55:40.731675 15430 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 51, Time since last communication: 0.000s
I20250624 07:55:40.734534 15435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52afd679ffb241a3a9022cc850c03493. Latest consensus state: current_term: 9 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:40.734719 15435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:40.736377 15430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:40.736573 15430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493 [sys.catalog]: This master's current role is: LEADER
I20250624 07:55:40.736634 15435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "52afd679ffb241a3a9022cc850c03493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } } }
I20250624 07:55:40.736794 15435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [sys.catalog]: This master's current role is: FOLLOWER
I20250624 07:55:40.756094 15432 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 07:55:40.758924 15432 catalog_manager.cc:671] Loaded metadata for table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542]
I20250624 07:55:40.759346 15432 catalog_manager.cc:671] Loaded metadata for table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2]
I20250624 07:55:40.759691 15432 catalog_manager.cc:671] Loaded metadata for table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b]
I20250624 07:55:40.760075 15432 catalog_manager.cc:671] Loaded metadata for table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939]
I20250624 07:55:40.760367 15432 catalog_manager.cc:671] Loaded metadata for table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e]
I20250624 07:55:40.760710 15432 catalog_manager.cc:671] Loaded metadata for table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72]
I20250624 07:55:40.761093 15432 catalog_manager.cc:671] Loaded metadata for table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e]
I20250624 07:55:40.761420 15432 catalog_manager.cc:671] Loaded metadata for table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62]
I20250624 07:55:40.761744 15432 catalog_manager.cc:671] Loaded metadata for table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d]
I20250624 07:55:40.763587 15432 tablet_loader.cc:96] loaded metadata for tablet 1d7e38715d1543c69f3ceee6fb74d54a (table default.table_b66cb6b1a255496bb4909de6a39e80fe [id=4cf33f8a99214ccb9579b1538db61939])
I20250624 07:55:40.764394 15432 tablet_loader.cc:96] loaded metadata for tablet 31617090f65045f08d94b793ff525275 (table default.table_0047240c5fdf4b85bc2b5fda6bdcef32 [id=1cefaef58d9f43afa6a3438f6ab5fdc2])
I20250624 07:55:40.765072 15432 tablet_loader.cc:96] loaded metadata for tablet 601669524f5447929fa9a18597f31d61 (table default.table_6e36abec7aef4670b05e620d91bba579 [id=dc29fa228afc408192657b4c7fe63a62])
I20250624 07:55:40.765437 15432 tablet_loader.cc:96] loaded metadata for tablet 66cf2fd1ecd74a2b9d9e098cea718c7d (table default.table_8a84bf94dbd94e65946d5e56b69172be [id=848ba893f1dd47beb81f7e789f7b333e])
I20250624 07:55:40.766084 15432 tablet_loader.cc:96] loaded metadata for tablet 819b10f76c2c40c4891d7bd802df5c4d (table default.table_02926d7fe80d4545a80eb082aa7d2671 [id=15f1d68245954151b947d4c05426f542])
I20250624 07:55:40.766546 15432 tablet_loader.cc:96] loaded metadata for tablet 8ede62aa66ba4b829be7c283993e3cf7 (table default.table_ba78959886d2470285dacaf16007267d [id=3aea8f0c52b94eb69eba1ec6142aba0b])
I20250624 07:55:40.766839 15432 tablet_loader.cc:96] loaded metadata for tablet 901ae0683a114638a6a075cf24c5eabe (table default.table_2974357596ca4893b46d7199b1367e2f [id=5320ede59ed2474087a59600656c509e])
I20250624 07:55:40.767087 15432 tablet_loader.cc:96] loaded metadata for tablet a7fc1a7fbc17491caa3b44e0a4db6590 (table default.table_953a2fe7f1694bb7b1afcbbaddd632b6 [id=ec5b6a77d9b44838b45fac61e863953d])
I20250624 07:55:40.767803 15432 tablet_loader.cc:96] loaded metadata for tablet f5af61fc5261444b8544d72aa61d3082 (table default.table_fc2c06ca3ddc4d78baaced0d6345b281 [id=58e77a9ccbcf4180acf57ae35e6f4b72])
I20250624 07:55:40.768023 15432 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 07:55:40.769016 15432 catalog_manager.cc:1261] Loaded cluster ID: 50daecf2c21c4ff1a05ae6bc8eec5957
I20250624 07:55:40.769083 15432 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 07:55:40.770300 15432 catalog_manager.cc:1506] Loading token signing keys...
I20250624 07:55:40.771095 15432 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 52afd679ffb241a3a9022cc850c03493: Loaded TSK: 0
I20250624 07:55:40.771867 15432 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 07:55:40.772698 15432 catalog_manager.cc:1529] Loading latest processed Hive Metastore notification log event ID...
I20250624 07:55:40.773514 15432 catalog_manager.cc:5804] Last processed Hive Metastore notification event ID: -1
F20250624 07:55:40.806854 15412 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_fc2c06ca3ddc4d78baaced0d6345b281, message={"server":"","servicePrincipal":"","db":"default","table":"table_fc2c06ca3ddc4d78baaced0d6345b281","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_fc2c06ca3ddc4d78baaced0d6345b281\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_fc2c06ca3ddc4d78baaced0d6345b281\",\"kudu.table_id\":\"58e77a9ccbcf4180acf57ae35e6f4b72\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1750751710, eventType=CREATE_TABLE, dbName=default, tableName=table_e87f394359f24830b4171bf6b60d4ab2, message={"server":"","servicePrincipal":"","db":"default","table":"table_e87f394359f24830b4171bf6b60d4ab2","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_e87f394359f24830b4171bf6b60d4ab2\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1750751710},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskPG5ypB/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1750751693136305-13008-0/minicluster-data/warehouse/table_e87f394359f24830b4171bf6b60d4ab2\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_e87f394359f24830b4171bf6b60d4ab2\",\"kudu.table_id\":\"ec5523acb9b448e0bf0c2cf085703c97\",\"transient_lastDdlTime\":\"1750751710\",\"comment\":\"\",\"kudu.master_addresses\":\"127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"50daecf2c21c4ff1a05ae6bc8eec5957\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1750751710,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f3383ffcdcd google::LogMessage::Fail() at ??:0
@ 0x7f3384000b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3383ffc7cc google::LogMessage::Flush() at ??:0
@ 0x7f3383ffca11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f339a3fd6d4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f339a3fbf81 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f339a403bfc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f339a403a79 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa7573d std::function<>::operator()() at ??:?
@ 0x7f3386d564fe kudu::Thread::SuperviseThread() at ??:0
@ 0x7f338a1746db start_thread at ??:0
@ 0x7f3382ead71f clone at ??:0
W20250624 07:55:41.005465 13820 connection.cc:537] client connection to 127.12.180.61:46565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250624 07:55:41.005887 13939 connection.cc:537] client connection to 127.12.180.61:46565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250624 07:55:41.133080 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:41.133236 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:41.236302 15435 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:41.236513 15435 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:41.237280 15435 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:41.238613 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:41.238752 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:41.238873 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:41.239117 15435 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:55:41.842087 15466 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:41.842319 15466 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:41.843029 15466 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:41.845642 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:41.845824 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:41.845973 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:41.846261 15466 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:55:42.171154 13008 master-stress-test.cc:577] Killing test thread 1/25
I20250624 07:55:42.736971 15475 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:42.737177 15475 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:42.737778 15475 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:42.738668 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250624 07:55:42.739991 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:42.740150 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:42.740259 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:42.740458 15475 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:55:43.986565 15487 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:43.986786 15487 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:43.987385 15487 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:43.989387 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:43.989578 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:43.989692 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:43.989948 15487 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:55:45.102629 15498 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:45.102849 15498 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:45.103422 15498 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:45.105180 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:45.105341 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:45.105461 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:45.105711 15498 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:55:46.136029 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:46.136232 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:55:48.116616 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 6: this message will repeat every 5th retry.
I20250624 07:55:48.482438 15519 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:48.482664 15519 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:48.483350 15519 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:48.484431 15183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250624 07:55:48.485837 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:55:48.486020 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:55:48.486146 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:48.486464 15519 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:55:50.406311 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 6: this message will repeat every 5th retry.
I20250624 07:55:51.138854 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:51.139020 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:55:54.102594 15557 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:55:54.102783 15557 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:55:54.103332 15557 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:55:54.104147 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:55:54.105419 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:55:54.105551 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:55:54.105726 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:55:54.105952 15557 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:55:56.144022 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:55:56.144215 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:55:56.861084 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 11: this message will repeat every 5th retry.
W20250624 07:56:00.098505 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 11: this message will repeat every 5th retry.
I20250624 07:56:01.146991 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:01.147166 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:56:02.411772 15614 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:56:02.411998 15614 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:56:02.412668 15614 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:56:02.413651 15183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:56:02.414575 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:56:02.414708 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:56:02.414820 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:56:02.415045 15614 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:56:06.149710 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:06.149860 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:56:06.403244 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 16: this message will repeat every 5th retry.
W20250624 07:56:10.244732 13659 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (86 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:56:10.249191 13796 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (58 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:56:10.548272 13930 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (97 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:56:10.801685 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 16: this message will repeat every 5th retry.
I20250624 07:56:11.052850 15694 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:56:11.053071 15694 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:56:11.053686 15694 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:56:11.055948 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:56:11.056998 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:56:11.057157 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:56:11.057266 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:56:11.057495 15694 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:56:11.152024 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:11.152195 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:56:15.919006 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 21: this message will repeat every 5th retry.
I20250624 07:56:16.154906 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:16.155074 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:56:20.700889 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 21: this message will repeat every 5th retry.
I20250624 07:56:21.157261 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:21.157407 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:56:22.016800 15816 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:56:22.017011 15816 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:56:22.017642 15816 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:56:22.018534 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:56:22.019570 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:56:22.019718 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:56:22.019816 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:56:22.020056 15816 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:56:26.053658 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 26: this message will repeat every 5th retry.
I20250624 07:56:26.159713 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:26.159888 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:56:30.121012 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 26: this message will repeat every 5th retry.
I20250624 07:56:31.162199 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:31.162379 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:56:36.164460 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:36.164614 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:56:36.424747 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 31: this message will repeat every 5th retry.
I20250624 07:56:38.758983 15993 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:56:38.759179 15993 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:56:38.759933 15993 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:56:38.760751 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:56:38.762018 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:56:38.762142 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:56:38.762295 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:56:38.762499 15993 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:56:40.122947 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 31: this message will repeat every 5th retry.
I20250624 07:56:41.167373 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:41.167531 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:56:46.169864 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:46.170042 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:56:46.241140 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 36: this message will repeat every 5th retry.
I20250624 07:56:46.650436 16088 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:56:46.650619 16088 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:56:46.651253 16088 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:56:46.651762 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:56:46.654960 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:56:46.655143 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:56:46.655252 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:56:46.655494 16088 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:56:50.345067 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 36: this message will repeat every 5th retry.
I20250624 07:56:51.172406 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:51.172564 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:56:54.975507 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 41: this message will repeat every 5th retry.
I20250624 07:56:56.174875 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:56:56.175031 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:00.621332 16247 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:57:00.621511 16247 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:57:00.622298 16247 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:57:00.623270 15183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:57:00.625005 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:57:00.625169 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:57:00.625277 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:57:00.625485 16247 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:57:00.867373 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 41: this message will repeat every 5th retry.
I20250624 07:57:01.177115 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:01.177266 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:57:05.433655 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 46: this message will repeat every 5th retry.
I20250624 07:57:06.179355 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:06.179514 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:57:10.407184 13794 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (214 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:57:10.571966 13657 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (317 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:57:10.671105 13930 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (217 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:57:11.102131 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 46: this message will repeat every 5th retry.
I20250624 07:57:11.181687 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:11.181847 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:57:15.872397 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 51: this message will repeat every 5th retry.
I20250624 07:57:16.184247 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:16.184414 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:18.797113 16453 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:57:18.797295 16453 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:57:18.798027 16453 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:57:18.799600 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:57:18.800598 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:57:18.800714 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:57:18.800834 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:57:18.801045 16453 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:57:20.176923 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 51: this message will repeat every 5th retry.
I20250624 07:57:21.186931 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:21.187083 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:23.912164 16534 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:57:23.912364 16534 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:57:23.913034 16534 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:57:23.913861 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:57:23.915035 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:57:23.915186 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:57:23.915306 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:57:23.915539 16534 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:57:26.189126 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:26.189301 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:57:26.875023 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 56: this message will repeat every 5th retry.
W20250624 07:57:30.499158 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 56: this message will repeat every 5th retry.
I20250624 07:57:31.191956 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:31.192118 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:36.151087 15188 maintenance_manager.cc:419] P 588e956d4ac040deb3c06e57bd3c6b88: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033351
I20250624 07:57:36.196146 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:36.196306 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:36.196514 15187 maintenance_manager.cc:643] P 588e956d4ac040deb3c06e57bd3c6b88: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.019s sys 0.004s Metrics: {"bytes_written":15035,"cfile_init":1,"dirs.queue_time_us":342,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":828,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":569,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":32,"thread_start_us":378,"threads_started":2,"wal-append.queue_time_us":248}
I20250624 07:57:36.197721 15188 maintenance_manager.cc:419] P 588e956d4ac040deb3c06e57bd3c6b88: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1750 bytes on disk
I20250624 07:57:36.198406 15187 maintenance_manager.cc:643] P 588e956d4ac040deb3c06e57bd3c6b88: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20250624 07:57:36.635825 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 61: this message will repeat every 5th retry.
W20250624 07:57:39.968756 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 61: this message will repeat every 5th retry.
I20250624 07:57:41.198913 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:41.199055 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:41.981909 16779 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:57:41.982087 16779 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:57:41.982890 16779 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:57:41.983664 15183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:57:41.985224 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:57:41.985357 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:57:41.985478 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:57:41.985685 16779 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:57:45.951751 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 66: this message will repeat every 5th retry.
I20250624 07:57:46.201438 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:46.201596 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:57:49.771435 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 66: this message will repeat every 5th retry.
I20250624 07:57:51.204380 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:51.204551 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:55.377128 17013 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:57:55.377317 17013 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:57:55.378029 17013 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:57:55.378929 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:57:55.379985 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:57:55.380173 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:57:55.380262 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:57:55.380512 17013 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:57:55.582732 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 71: this message will repeat every 5th retry.
I20250624 07:57:56.207146 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:57:56.207337 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:57:56.568315 17042 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:57:56.568493 17042 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:57:56.569016 17042 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:57:56.570256 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:57:56.570386 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:57:56.570518 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:57:56.570741 17042 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:58:00.050132 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 71: this message will repeat every 5th retry.
I20250624 07:58:01.209618 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:01.209767 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:04.930523 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 76: this message will repeat every 5th retry.
I20250624 07:58:06.212289 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:06.212445 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:58:08.341531 17236 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:58:08.341701 17236 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:58:08.342343 17236 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:58:08.343147 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250624 07:58:08.344062 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:58:08.344221 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:58:08.344341 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:58:08.344550 17236 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:58:09.359136 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 76: this message will repeat every 5th retry.
W20250624 07:58:10.565527 13794 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (334 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:58:10.565537 13796 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (298 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:58:10.640261 13659 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (457 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:58:10.833258 13932 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (299 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:58:11.214805 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:11.214982 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:58:14.579172 17351 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:58:14.579349 17351 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:58:14.579887 17351 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:58:14.580797 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:58:14.581782 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:58:14.581889 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:58:14.582015 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:58:14.582265 17351 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:58:15.200528 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 81: this message will repeat every 5th retry.
I20250624 07:58:16.217612 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:16.217767 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:19.332664 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 81: this message will repeat every 5th retry.
I20250624 07:58:21.220574 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:21.220781 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:25.255093 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 86: this message will repeat every 5th retry.
I20250624 07:58:25.435771 17539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:58:25.435966 17539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:58:25.436556 17539 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:58:25.437914 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:58:25.438961 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:58:25.439086 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:58:25.439205 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:58:25.439416 17539 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:58:26.223377 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:26.223538 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:28.811055 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 86: this message will repeat every 5th retry.
I20250624 07:58:31.226094 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:31.226281 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:34.553164 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 91: this message will repeat every 5th retry.
I20250624 07:58:36.228971 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:36.229141 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:38.789855 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 91: this message will repeat every 5th retry.
I20250624 07:58:39.544713 17785 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:58:39.544905 17785 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:58:39.545532 17785 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:58:39.546489 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:58:39.547526 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:58:39.547675 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:58:39.547786 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:58:39.547964 17785 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:58:41.231541 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:41.231714 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:43.989812 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 96: this message will repeat every 5th retry.
I20250624 07:58:46.234521 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:46.234678 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:49.381964 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 96: this message will repeat every 5th retry.
I20250624 07:58:50.038203 17968 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:58:50.038394 17968 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:58:50.038957 17968 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:58:50.039875 15183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:58:50.040894 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:58:50.041083 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:58:50.041169 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:58:50.041371 17968 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:58:51.237071 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:51.237244 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:54.035208 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 101: this message will repeat every 5th retry.
I20250624 07:58:56.239745 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:58:56.239923 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:58:59.287241 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 101: this message will repeat every 5th retry.
I20250624 07:59:01.242789 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:01.242990 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:04.140131 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 106: this message will repeat every 5th retry.
I20250624 07:59:04.407029 18223 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:59:04.407204 18223 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:59:04.407806 18223 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:59:04.408643 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:59:04.409950 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:59:04.410147 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:59:04.410287 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:59:04.410506 18223 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:59:06.245224 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:06.245380 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:09.585786 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 106: this message will repeat every 5th retry.
W20250624 07:59:10.727984 13794 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (454 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:59:10.808537 13659 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (642 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:59:10.995291 13932 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (419 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:59:11.248199 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:11.248386 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:14.315975 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 111: this message will repeat every 5th retry.
I20250624 07:59:14.992408 18409 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:59:14.992620 18409 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:59:14.993261 18409 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:59:14.994038 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:59:14.995339 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:59:14.995486 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:59:14.995591 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:59:14.995790 18409 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:59:16.250777 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:16.250950 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:19.314159 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 111: this message will repeat every 5th retry.
I20250624 07:59:21.253458 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:21.253626 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:24.039083 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 116: this message will repeat every 5th retry.
I20250624 07:59:26.256255 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:26.256407 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:59:27.650466 18637 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:59:27.650664 18637 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:59:27.651304 18637 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:59:27.652148 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:59:27.653126 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:59:27.653242 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:59:27.653872 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:59:27.654093 18637 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:59:30.172050 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 116: this message will repeat every 5th retry.
I20250624 07:59:31.259151 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:31.259307 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:34.107982 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 121: this message will repeat every 5th retry.
I20250624 07:59:36.261852 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:36.262032 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:39.284891 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 121: this message will repeat every 5th retry.
I20250624 07:59:41.264492 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:41.264652 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:59:41.416496 18881 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:59:41.416672 18881 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:59:41.417369 18881 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:59:41.417917 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 07:59:41.419026 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:59:41.419258 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:59:41.419387 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:59:41.419580 18881 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:59:42.141675 18899 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:59:42.141851 18899 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:59:42.142459 18899 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:59:42.144287 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:59:42.144485 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:59:42.144631 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:59:42.144843 18899 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:59:44.872557 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 126: this message will repeat every 5th retry.
I20250624 07:59:46.267120 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:46.267272 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:49.361894 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 126: this message will repeat every 5th retry.
I20250624 07:59:51.269869 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:51.270031 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 07:59:52.063416 19070 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:59:52.063596 19070 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:59:52.064288 19070 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:59:52.065318 15183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250624 07:59:52.066321 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 07:59:52.066471 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 07:59:52.066579 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:59:52.066780 19070 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 07:59:53.953935 19106 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 07:59:53.954093 19106 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 07:59:53.954680 19106 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 07:59:53.956494 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 07:59:53.956635 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 07:59:53.956748 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 07:59:53.956965 19106 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 07:59:54.900903 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 131: this message will repeat every 5th retry.
I20250624 07:59:56.272657 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 07:59:56.272821 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 07:59:59.990916 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 131: this message will repeat every 5th retry.
I20250624 08:00:01.275220 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:01.275409 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 08:00:04.713699 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 136: this message will repeat every 5th retry.
I20250624 08:00:06.277622 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:06.277773 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 08:00:07.243338 19333 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 08:00:07.243512 19333 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 08:00:07.244081 19333 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 08:00:07.245622 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250624 08:00:07.246752 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 08:00:07.246905 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 08:00:07.247004 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 08:00:07.247225 19333 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 08:00:09.692663 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 136: this message will repeat every 5th retry.
W20250624 08:00:10.893537 13794 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (574 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 08:00:11.158288 13930 heartbeater.cc:646] Failed to heartbeat to 127.12.180.60:33449 (577 consecutive failures): Network error: Failed to ping master at 127.12.180.60:33449: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 08:00:11.280113 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:11.280272 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 08:00:11.296963 13659 heartbeater.cc:646] Failed to heartbeat to 127.12.180.61:46565 (829 consecutive failures): Network error: Failed to ping master at 127.12.180.61:46565: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 08:00:14.478477 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 141: this message will repeat every 5th retry.
I20250624 08:00:16.282805 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:16.282961 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 08:00:19.767225 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 141: this message will repeat every 5th retry.
I20250624 08:00:21.285193 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:21.285357 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 08:00:24.439936 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 146: this message will repeat every 5th retry.
I20250624 08:00:24.669204 19615 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 08:00:24.669399 19615 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 08:00:24.670260 19615 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 08:00:24.671124 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 08:00:24.672374 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 08:00:24.672544 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
I20250624 08:00:24.672652 15183 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 08:00:24.672870 19615 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250624 08:00:26.287668 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:26.287865 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 08:00:30.118157 13545 consensus_peers.cc:487] T 5daee63b211a4b099533045210d213c8 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5daee63b211a4b099533045210d213c8. This is attempt 146: this message will repeat every 5th retry.
I20250624 08:00:31.291024 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:31.291198 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250624 08:00:34.788122 13545 consensus_peers.cc:487] T 2fc78118d4fa41d0b4a1aabb92616335 P dbb2615829f748ef929761ed90057cb7 -> Peer ab43ace4a8b44b908d247cad235e06df (127.12.180.2:43137): Couldn't send request to peer ab43ace4a8b44b908d247cad235e06df. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc78118d4fa41d0b4a1aabb92616335. This is attempt 151: this message will repeat every 5th retry.
I20250624 08:00:36.293362 15163 master_runner.cc:186] Error getting master registration for 127.12.180.61:46565: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111),
I20250624 08:00:36.293519 15163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250624 08:00:36.695432 15188 maintenance_manager.cc:419] P 588e956d4ac040deb3c06e57bd3c6b88: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1750 bytes on disk
I20250624 08:00:36.699883 15187 maintenance_manager.cc:643] P 588e956d4ac040deb3c06e57bd3c6b88: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.001s sys 0.003s Metrics: {"dirs.queue_time_us":385,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":92,"thread_start_us":468,"threads_started":2,"wal-append.queue_time_us":306}
I20250624 08:00:36.924746 19808 raft_consensus.cc:491] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52afd679ffb241a3a9022cc850c03493)
I20250624 08:00:36.924939 19808 raft_consensus.cc:513] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "588e956d4ac040deb3c06e57bd3c6b88" member_type: VOTER last_known_addr { host: "127.12.180.62" port: 44637 } } peers { permanent_uuid: "52afd679ffb241a3a9022cc850c03493" member_type: VOTER last_known_addr { host: "127.12.180.61" port: 46565 } } peers { permanent_uuid: "632ba412a681447e93af32d7fdf0691d" member_type: VOTER last_known_addr { host: "127.12.180.60" port: 33449 } }
I20250624 08:00:36.926148 19808 leader_election.cc:290] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565), 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449)
W20250624 08:00:36.926746 15183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250624 08:00:36.927867 15183 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 632ba412a681447e93af32d7fdf0691d (127.12.180.60:33449): Network error: Client connection negotiation failed: client connection to 127.12.180.60:33449: connect: Connection refused (error 111)
W20250624 08:00:36.928020 15185 leader_election.cc:336] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 52afd679ffb241a3a9022cc850c03493 (127.12.180.61:46565): Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
I20250624 08:00:36.928115 15185 leader_election.cc:304] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 588e956d4ac040deb3c06e57bd3c6b88; no voters: 52afd679ffb241a3a9022cc850c03493, 632ba412a681447e93af32d7fdf0691d
I20250624 08:00:36.928337 19808 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 588e956d4ac040deb3c06e57bd3c6b88 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250624 08:00:37.154771 13940 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 08:00:37.155022 13940 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 08:00:37.155136 13940 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
W20250624 08:00:37.155236 13940 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
F20250624 08:00:37.167026 13955 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1750752037 (unix time) try "date -d @1750752037" if you are using GNU date ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000032d0) received by PID 13008 (TID 0x7f69392aa700) from PID 13008; stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f6950c57980 (unknown) at ??:0
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175050 13953 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_26071aa0d9c8459b8f1d9583dedee21a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175050 13953 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_26071aa0d9c8459b8f1d9583dedee21a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175067 13950 master-stress-test.cc:328] Check failed: _s.ok() Bad status: F20250624 08:00:37.175136 13959 out: Error creating table default.Table_17819dd7286645d8a22ec7eed153297a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175050 13953 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_26071aa0d9c8459b8f1d9583dedee21a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175067 13950 master-stress-test.cc:328] Check failed: _s.ok() Bad status: F20250624 08:00:37.175136 13959 out: Error creating table default.Table_17819dd7286645d8a22ec7eed153297a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175050 13953 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_26071aa0d9c8459b8f1d9583dedee21a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175067 13950 master-stress-test.cc:328] Check failed: _s.ok() Bad status: F20250624 08:00:37.175136 13959 out: Error creating table default.Table_17819dd7286645d8a22ec7eed153297a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175323 13945 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ec6cab3afe9e48f3a38fa9c0c2ca3922 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175050 13953 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_26071aa0d9c8459b8f1d9583dedee21a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175067 13950 master-stress-test.cc:328] Check failed: _s.ok() Bad status: F20250624 08:00:37.175136 13959 out: Error creating table default.Table_17819dd7286645d8a22ec7eed153297a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175323 13945 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ec6cab3afe9e48f3a38fa9c0c2ca3922 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175487 13956 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175050 13953 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_26071aa0d9c8459b8f1d9583dedee21a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs:*** Check failure stack trace: ***
@ 0x7f6949c31fb7 gsignal at ??:0
127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175067 13950 master-stress-test.cc:328] Check failed: _s.ok() Bad status: F20250624 08:00:37.175136 13959 out: Error creating table default.Table_17819dd7286645d8a22ec7eed153297a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175323 13945 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ec6cab3afe9e48f3a38fa9c0c2ca3922 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175487 13956 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175664 13962 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20250624 08:00:37.168144 13960 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.168242 13948 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_6c5708ec0a8a44f4bc2b8d669d041a08 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169435 13971 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169485 13966 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.169582 13946 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_50b9eb18e89f4d0a9d1b64950157ffe4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170665 13958 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170840 13947 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c6ab687a1f6d4a9a93bc3f1eee4c8b19 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.170930 13970 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171005 13965 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.171108 13951 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_fe124062a0864c5e9472552b1a471845 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.172175 13949 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c316e293c95c47eda371175ca862fe0d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173285 13957 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173463 13954 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3e714bb6900c450b90b4519baaaa3aba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173480 13963 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.173664 13961 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.174703 13952 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_258e5344814340e49754e36a4631c5e0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175050 13953 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_26071aa0d9c8459b8f1d9583dedee21a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175067 13950 master-stress-test.cc:328] Check failed: _s.ok() Bad status: F20250624 08:00:37.175136 13959 out: Error creating table default.Table_17819dd7286645d8a22ec7eed153297a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111) master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175323 13945 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ec6cab3afe9e48f3a38fa9c0c2ca3922 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175487 13956 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175664 13962 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)F20250624 08:00:37.175706 13969 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.12.180.62:44637,127.12.180.61:46565,127.12.180.60:33449, num_attempts: 770) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.12.180.61:46565: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f6949c33921 abort at ??:0
@ 0x7f694c969dcd google::LogMessage::Fail() at ??:0
@ 0x7f694c96db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f694c9697cc google::LogMessage::Flush() at ??:0
@ 0x7f694c96af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x409911 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:328
@ 0x447fd4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x447e8d std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x447e4a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x447dfe std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x447b39 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f694a2b96df (unknown) at ??:0
@ 0x7f6950c4c6db start_thread at ??:0
@ 0x7f6949d1471f clone at ??:0