Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251213 20:58:52.824311 11708 test_util.cc:276] Using random seed: -1522561338
2025-12-13T20:58:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-13T20:58:52Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskli_hB8/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskli_hB8/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/security.properties
2025-12-13 20:58:59: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskli_hB8/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskli_hB8/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20251213 20:59:08.212915 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.62:33723
--webserver_interface=127.11.111.62
--webserver_port=0
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251213 20:59:08.531762 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:08.532111 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:08.532234 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:08.540410 11951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:08.540534 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:08.540612 11951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:08.540674 11951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:08.550488 11951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.62:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.111.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:08.552148 11951 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:08.554020 11951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:08.562316 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:08.562398 11957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:08.562834 11959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:08.563459 11951 server_base.cc:1047] running on GCE node
I20251213 20:59:08.564188 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:08.565716 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:08.566916 11951 hybrid_clock.cc:648] HybridClock initialized: now 1765659548566861 us; error 44 us; skew 500 ppm
I20251213 20:59:08.567286 11951 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:08.569630 11951 webserver.cc:492] Webserver started at http://127.11.111.62:44033/ using document root <none> and password file <none>
I20251213 20:59:08.570396 11951 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:08.570495 11951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:08.570844 11951 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251213 20:59:08.573263 11951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/instance:
uuid: "f2fb6eff9fdc4139b18baa7e29377276"
format_stamp: "Formatted at 2025-12-13 20:59:08 on dist-test-slave-p3t2"
I20251213 20:59:08.574056 11951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal/instance:
uuid: "f2fb6eff9fdc4139b18baa7e29377276"
format_stamp: "Formatted at 2025-12-13 20:59:08 on dist-test-slave-p3t2"
I20251213 20:59:08.579676 11951 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251213 20:59:08.583168 11965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:08.584590 11951 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251213 20:59:08.584771 11951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
uuid: "f2fb6eff9fdc4139b18baa7e29377276"
format_stamp: "Formatted at 2025-12-13 20:59:08 on dist-test-slave-p3t2"
I20251213 20:59:08.584976 11951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:08.607458 11951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:08.608361 11951 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:08.608626 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:08.627552 11951 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.62:33723
I20251213 20:59:08.627596 12016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.62:33723 every 8 connection(s)
I20251213 20:59:08.629235 11951 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/info.pb
I20251213 20:59:08.633644 12017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:08.638607 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 11951
I20251213 20:59:08.638882 11708 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal/instance
I20251213 20:59:08.639539 12017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20251213 20:59:08.641494 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--webserver_interface=127.11.111.61
--webserver_port=0
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251213 20:59:08.659804 12017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } has no permanent_uuid. Determining permanent_uuid...
W20251213 20:59:08.660869 11969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:08.662398 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:08.711742 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } attempt: 1
W20251213 20:59:08.713893 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:08.782146 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } attempt: 2
W20251213 20:59:08.783994 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:08.875229 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } attempt: 3
W20251213 20:59:08.876868 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:08.968742 12020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:08.969054 12020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:08.969159 12020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:08.977310 12020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:08.977425 12020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:08.977491 12020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:08.977551 12020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:08.987421 12020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.111.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:08.989014 12020 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:08.990861 12020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:08.999172 12026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:08.999214 12027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:08.999593 12029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:09.000046 12020 server_base.cc:1047] running on GCE node
I20251213 20:59:09.000830 12020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:09.002282 12020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:09.003500 12020 hybrid_clock.cc:648] HybridClock initialized: now 1765659549003451 us; error 38 us; skew 500 ppm
I20251213 20:59:09.003836 12020 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:09.005971 12020 webserver.cc:492] Webserver started at http://127.11.111.61:42211/ using document root <none> and password file <none>
I20251213 20:59:09.006695 12020 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:09.006794 12020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:09.007133 12020 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251213 20:59:09.009491 12020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/instance:
uuid: "68c0fc24cab047dcaa4cae2dbd628626"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:09.010242 12020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal/instance:
uuid: "68c0fc24cab047dcaa4cae2dbd628626"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:09.015425 12020 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20251213 20:59:09.018764 12035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:09.020040 12020 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251213 20:59:09.020097 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } attempt: 4
I20251213 20:59:09.020188 12020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
uuid: "68c0fc24cab047dcaa4cae2dbd628626"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:09.020406 12020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251213 20:59:09.021880 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:09.046954 12020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:09.047916 12020 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:09.048149 12020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:09.066778 12020 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.61:39203
I20251213 20:59:09.066797 12086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.61:39203 every 8 connection(s)
I20251213 20:59:09.068275 12020 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
I20251213 20:59:09.072527 12087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:09.077991 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 12020
I20251213 20:59:09.078173 11708 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal/instance
I20251213 20:59:09.078735 12087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20251213 20:59:09.079547 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--webserver_interface=127.11.111.60
--webserver_port=0
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251213 20:59:09.093832 12087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } has no permanent_uuid. Determining permanent_uuid...
I20251213 20:59:09.102748 12087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
W20251213 20:59:09.103768 12038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:09.105221 12087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:09.154603 12087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 1
W20251213 20:59:09.156523 12087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:09.224810 12087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 2
W20251213 20:59:09.226732 12087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:09.317987 12087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 3
W20251213 20:59:09.319870 12087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:09.321080 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } attempt: 5
I20251213 20:59:09.326917 12017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
W20251213 20:59:09.328478 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:09.379689 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 1
W20251213 20:59:09.381455 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:09.407734 12089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:09.408043 12089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:09.408155 12089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:09.416374 12089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:09.416471 12089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:09.416539 12089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:09.416603 12089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:09.426723 12089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.111.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:09.428545 12089 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:09.430431 12089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:09.438511 12097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:09.438361 12099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:09.438560 12096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:09.439443 12089 server_base.cc:1047] running on GCE node
I20251213 20:59:09.440176 12089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:09.441640 12089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:09.442864 12089 hybrid_clock.cc:648] HybridClock initialized: now 1765659549442793 us; error 53 us; skew 500 ppm
I20251213 20:59:09.443220 12089 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:09.445444 12089 webserver.cc:492] Webserver started at http://127.11.111.60:36499/ using document root <none> and password file <none>
I20251213 20:59:09.446223 12089 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:09.446324 12089 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:09.446671 12089 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251213 20:59:09.449003 12089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/instance:
uuid: "6736d1c95f384b9098c089a20b76e9ae"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:09.449664 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 2
I20251213 20:59:09.449965 12089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal/instance:
uuid: "6736d1c95f384b9098c089a20b76e9ae"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
W20251213 20:59:09.451395 12017 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:09.455498 12089 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251213 20:59:09.458722 12105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:09.460068 12089 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251213 20:59:09.460245 12089 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
uuid: "6736d1c95f384b9098c089a20b76e9ae"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:09.460444 12089 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:09.463112 12087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 4
W20251213 20:59:09.464855 12087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:09.481665 12089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:09.482547 12089 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:09.482791 12089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:09.500651 12089 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.60:36039
I20251213 20:59:09.500659 12156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.60:36039 every 8 connection(s)
I20251213 20:59:09.502118 12089 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
I20251213 20:59:09.506249 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 12089
I20251213 20:59:09.506381 12157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:09.506433 11708 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal/instance
I20251213 20:59:09.512048 12157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } has no permanent_uuid. Determining permanent_uuid...
I20251213 20:59:09.537153 12157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } has no permanent_uuid. Determining permanent_uuid...
I20251213 20:59:09.543828 12157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } has no permanent_uuid. Determining permanent_uuid...
I20251213 20:59:09.553695 12157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap starting.
I20251213 20:59:09.557116 12157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:09.557602 12017 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 3
I20251213 20:59:09.558429 12157 log.cc:826] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:09.561285 12157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: No bootstrap required, opened a new log
I20251213 20:59:09.567397 12157 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.567696 12157 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:09.567845 12157 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Initialized, Role: FOLLOWER
I20251213 20:59:09.568157 12017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Bootstrap starting.
I20251213 20:59:09.568583 12157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.570011 12163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.570297 12163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:09.571055 12157 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:09.572180 12017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:09.573458 12017 log.cc:826] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:09.576452 12017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: No bootstrap required, opened a new log
W20251213 20:59:09.584937 12178 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20251213 20:59:09.585346 12017 raft_consensus.cc:359] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.585728 12017 raft_consensus.cc:385] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:09.585862 12017 raft_consensus.cc:740] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fb6eff9fdc4139b18baa7e29377276, State: Initialized, Role: FOLLOWER
I20251213 20:59:09.586719 12017 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
W20251213 20:59:09.585191 12178 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251213 20:59:09.585186 12089 master_runner.cc:186] Error getting master registration for 127.11.111.62:33723: OK, instance_id { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" instance_seqno: 1765659548622302 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251213 20:59:09.591305 12089 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:09.594009 12181 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.594287 12181 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:09.595459 12017 sys_catalog.cc:565] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: configured and running, proceeding with master startup.
W20251213 20:59:09.611403 11971 tablet.cc:2378] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251213 20:59:09.613193 11951 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: OK, instance_id { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" instance_seqno: 1765659549061752 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251213 20:59:09.623545 11951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251213 20:59:09.623303 12196 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251213 20:59:09.624029 12196 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251213 20:59:09.723626 12163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:09.723945 12163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.725867 12163 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:09.726945 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
I20251213 20:59:09.727490 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 0.
I20251213 20:59:09.728158 12107 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:09.728637 12163 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:09.728799 12163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:09.728906 12163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:09.731701 12163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.732771 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276"
I20251213 20:59:09.732817 12163 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 1 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:09.732965 11992 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 0 FOLLOWER]: Advancing to term 1
W20251213 20:59:09.734759 12111 tablet.cc:2378] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251213 20:59:09.735915 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 1.
I20251213 20:59:09.736567 12107 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:09.736889 12163 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:09.758014 12163 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 1 LEADER]: Becoming Leader. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Running, Role: LEADER
I20251213 20:59:09.758711 12163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.764042 12087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } attempt: 5
I20251213 20:59:09.777666 12087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap starting.
I20251213 20:59:09.781920 12087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:09.787038 12087 log.cc:826] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:09.792330 12087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: No bootstrap required, opened a new log
I20251213 20:59:09.793797 12198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 1 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.794173 12198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:09.795518 12201 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:09.799984 12201 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:09.803419 12087 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.803905 12087 raft_consensus.cc:385] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:09.804078 12087 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Initialized, Role: FOLLOWER
I20251213 20:59:09.804993 12087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:09.818061 12202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.818552 12202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:09.819648 12087 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:09.819682 12061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626"
I20251213 20:59:09.820142 12061 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:09.823263 12061 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 1.
I20251213 20:59:09.823639 12062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:09.823894 12062 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 1. Re-sending same reply.
I20251213 20:59:09.826749 12062 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Refusing update from remote peer 6736d1c95f384b9098c089a20b76e9ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251213 20:59:09.826933 11992 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 1 FOLLOWER]: Refusing update from remote peer 6736d1c95f384b9098c089a20b76e9ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251213 20:59:09.827764 12198 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:09.828591 12163 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:09.855012 12181 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 1 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.855350 12181 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:09.861822 12163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.862124 12163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:09.864583 12203 mvcc.cc:204] Tried to move back new op lower bound from 7232141516079992832 to 7232141515818303488. Current Snapshot: MvccSnapshot[applied={T|T < 7232141516079992832}]
I20251213 20:59:09.864871 12202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 1 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.865137 12202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:09.866467 12198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.866720 12198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:09.868072 12181 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.868307 12201 catalog_manager.cc:1357] Generated new cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:09.868285 12181 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:09.868403 12201 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:09.888414 12202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:09.888674 12202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:09.896993 12201 catalog_manager.cc:1380] Generated new certificate authority record
I20251213 20:59:09.900070 12201 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:09.918125 12201 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Generated new TSK 0
I20251213 20:59:09.919361 12201 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:09.920425 12201 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:09.921336 12201 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:09.932627 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.111.1:0
--local_ip_for_outbound_sockets=127.11.111.1
--webserver_interface=127.11.111.1
--webserver_port=0
--tserver_master_addrs=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251213 20:59:09.993445 12227 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:09.993671 12227 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: loading cluster ID for follower catalog manager: success
I20251213 20:59:09.997874 12227 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: acquiring CA information for follower catalog manager: success
I20251213 20:59:10.000754 12227 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:10.419245 12232 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:10.419564 12232 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
W20251213 20:59:10.421607 12223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20251213 20:59:10.421836 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
W20251213 20:59:10.421995 12223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20251213 20:59:10.422052 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c0fc24cab047dcaa4cae2dbd628626 in term 1.
W20251213 20:59:10.422125 12223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251213 20:59:10.422152 12132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6736d1c95f384b9098c089a20b76e9ae" is_pre_election: true
I20251213 20:59:10.422597 12037 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:10.423300 12232 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:10.423579 12232 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251213 20:59:10.423756 12232 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:10.423928 12232 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 1 FOLLOWER]: Advancing to term 2
I20251213 20:59:10.427711 12232 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:10.429117 12132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6736d1c95f384b9098c089a20b76e9ae"
I20251213 20:59:10.429817 12232 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 2 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:10.426728 12233 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:10.430593 12233 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:10.430991 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276"
I20251213 20:59:10.431274 11992 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 1 FOLLOWER]: Advancing to term 2
I20251213 20:59:10.432281 12233 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:10.433182 12132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6736d1c95f384b9098c089a20b76e9ae" is_pre_election: true
I20251213 20:59:10.434126 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c0fc24cab047dcaa4cae2dbd628626 in term 2.
I20251213 20:59:10.434403 12062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:10.434496 12037 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae
I20251213 20:59:10.434659 12062 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb6eff9fdc4139b18baa7e29377276 in current term 2: Already voted for candidate 68c0fc24cab047dcaa4cae2dbd628626 in this term.
W20251213 20:59:10.434990 12223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20251213 20:59:10.435031 12234 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 2 FOLLOWER]: Leader election won for term 2
I20251213 20:59:10.435031 11969 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
W20251213 20:59:10.435212 12223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.111.1
I20251213 20:59:10.435194 12234 raft_consensus.cc:697] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 2 LEADER]: Becoming Leader. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Running, Role: LEADER
I20251213 20:59:10.435717 12233 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251213 20:59:10.435704 12234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:10.437357 12234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c0fc24cab047dcaa4cae2dbd628626. Latest consensus state: current_term: 2 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:10.437709 12234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:10.438766 12235 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:10.441668 12235 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:10.442768 12235 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:10.442858 12235 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:10.444484 12235 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:10.446054 12235 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Loaded TSK: 0
I20251213 20:59:10.447368 12235 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:10.448446 12235 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:10.449126 12223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.111.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.111.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.111.1
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:10.449762 12235 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:10.450906 12223 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:10.452966 12223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251213 20:59:10.459491 11992 raft_consensus.cc:1240] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Rejecting Update request from peer 6736d1c95f384b9098c089a20b76e9ae for earlier term 1. Current term is 2. Ops: []
I20251213 20:59:10.460271 12236 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251213 20:59:10.461364 12238 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 1 LEADER]: Stepping down as leader of term 1
I20251213 20:59:10.461510 12238 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Running, Role: LEADER
I20251213 20:59:10.461863 12238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:10.462461 12238 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 1 FOLLOWER]: Advancing to term 2
W20251213 20:59:10.468506 12240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:10.468547 12239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:10.469336 12242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:10.470526 12223 server_base.cc:1047] running on GCE node
I20251213 20:59:10.471277 12223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:10.472641 12223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:10.473886 12223 hybrid_clock.cc:648] HybridClock initialized: now 1765659550473837 us; error 39 us; skew 500 ppm
I20251213 20:59:10.474284 12223 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:10.476939 12223 webserver.cc:492] Webserver started at http://127.11.111.1:42985/ using document root <none> and password file <none>
I20251213 20:59:10.477972 12223 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:10.478103 12223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:10.478588 12223 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251213 20:59:10.481952 12223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data/instance:
uuid: "97f3d76e88294646bccacef46bb93c38"
format_stamp: "Formatted at 2025-12-13 20:59:10 on dist-test-slave-p3t2"
I20251213 20:59:10.483106 12223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal/instance:
uuid: "97f3d76e88294646bccacef46bb93c38"
format_stamp: "Formatted at 2025-12-13 20:59:10 on dist-test-slave-p3t2"
I20251213 20:59:10.490257 12223 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20251213 20:59:10.494200 12248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:10.495762 12223 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251213 20:59:10.495936 12223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal
uuid: "97f3d76e88294646bccacef46bb93c38"
format_stamp: "Formatted at 2025-12-13 20:59:10 on dist-test-slave-p3t2"
I20251213 20:59:10.496124 12223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:10.531462 12223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:10.532299 12223 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:10.532560 12223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:10.533913 12223 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251213 20:59:10.535661 12223 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251213 20:59:10.535744 12223 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:10.535847 12223 ts_tablet_manager.cc:616] Registered 0 tablets
I20251213 20:59:10.535919 12223 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:10.570806 12223 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.1:36681
I20251213 20:59:10.570858 12360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.1:36681 every 8 connection(s)
I20251213 20:59:10.572220 12223 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/data/info.pb
I20251213 20:59:10.581243 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 12223
I20251213 20:59:10.581490 11708 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal/instance
I20251213 20:59:10.590376 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.111.2:0
--local_ip_for_outbound_sockets=127.11.111.2
--webserver_interface=127.11.111.2
--webserver_port=0
--tserver_master_addrs=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251213 20:59:10.593333 12178 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:10.624322 12362 heartbeater.cc:344] Connected to a master server at 127.11.111.62:33723
I20251213 20:59:10.624374 12361 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:10.624699 12362 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:10.624696 12361 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:10.625885 12361 heartbeater.cc:507] Master 127.11.111.60:36039 requested a full tablet report, sending...
I20251213 20:59:10.626147 12196 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:10.626216 12196 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: loading cluster ID for follower catalog manager: success
I20251213 20:59:10.627126 12362 heartbeater.cc:507] Master 127.11.111.62:33723 requested a full tablet report, sending...
I20251213 20:59:10.629002 11981 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:10.629690 12196 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: acquiring CA information for follower catalog manager: success
I20251213 20:59:10.631078 12122 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:10.632902 12196 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:10.634631 12363 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:10.634769 12363 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:10.635187 12363 heartbeater.cc:507] Master 127.11.111.61:39203 requested a full tablet report, sending...
I20251213 20:59:10.636585 12052 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:10.638187 12052 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.11.111.1:47461
I20251213 20:59:10.858543 11992 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Refusing update from remote peer 68c0fc24cab047dcaa4cae2dbd628626: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251213 20:59:10.859328 12234 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251213 20:59:10.861012 12233 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c0fc24cab047dcaa4cae2dbd628626. Latest consensus state: current_term: 2 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:10.861250 12233 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:10.863168 12232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:10.863437 12232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:10.863927 12132 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 2 FOLLOWER]: Refusing update from remote peer 68c0fc24cab047dcaa4cae2dbd628626: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251213 20:59:10.864705 12232 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251213 20:59:10.864773 12233 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:10.864931 12233 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:10.867049 12236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:10.867301 12236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:10.867084 12238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c0fc24cab047dcaa4cae2dbd628626. Latest consensus state: current_term: 2 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:10.867452 12238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
W20251213 20:59:10.968156 12371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:10.968602 12371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:10.968755 12371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:10.982816 12371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:10.983151 12371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.111.2
I20251213 20:59:10.999887 12371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.111.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.111.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.111.2
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:11.002388 12371 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:11.004930 12371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:11.016948 12380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:11.016949 12379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:11.018611 12382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:11.020401 12371 server_base.cc:1047] running on GCE node
I20251213 20:59:11.021063 12371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:11.022120 12371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:11.023353 12371 hybrid_clock.cc:648] HybridClock initialized: now 1765659551023280 us; error 62 us; skew 500 ppm
I20251213 20:59:11.023713 12371 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:11.026100 12371 webserver.cc:492] Webserver started at http://127.11.111.2:35227/ using document root <none> and password file <none>
I20251213 20:59:11.026902 12371 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:11.026996 12371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:11.027369 12371 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251213 20:59:11.029870 12371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data/instance:
uuid: "1b75487eefe54cbdb4140dea8ab8a499"
format_stamp: "Formatted at 2025-12-13 20:59:11 on dist-test-slave-p3t2"
I20251213 20:59:11.030689 12371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal/instance:
uuid: "1b75487eefe54cbdb4140dea8ab8a499"
format_stamp: "Formatted at 2025-12-13 20:59:11 on dist-test-slave-p3t2"
I20251213 20:59:11.036247 12371 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20251213 20:59:11.040416 12388 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:11.041932 12371 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251213 20:59:11.042093 12371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal
uuid: "1b75487eefe54cbdb4140dea8ab8a499"
format_stamp: "Formatted at 2025-12-13 20:59:11 on dist-test-slave-p3t2"
I20251213 20:59:11.042284 12371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:11.065533 12371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:11.067193 12371 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:11.067504 12371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:11.068662 12371 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251213 20:59:11.070878 12371 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251213 20:59:11.070989 12371 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:11.071097 12371 ts_tablet_manager.cc:616] Registered 0 tablets
I20251213 20:59:11.071168 12371 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:11.107175 12371 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.2:40023
I20251213 20:59:11.107239 12500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.2:40023 every 8 connection(s)
I20251213 20:59:11.108609 12371 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/data/info.pb
I20251213 20:59:11.112411 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 12371
I20251213 20:59:11.112604 11708 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal/instance
I20251213 20:59:11.116506 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.111.3:0
--local_ip_for_outbound_sockets=127.11.111.3
--webserver_interface=127.11.111.3
--webserver_port=0
--tserver_master_addrs=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251213 20:59:11.134994 12502 heartbeater.cc:344] Connected to a master server at 127.11.111.62:33723
I20251213 20:59:11.135385 12502 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:11.136552 12502 heartbeater.cc:507] Master 127.11.111.62:33723 requested a full tablet report, sending...
I20251213 20:59:11.136976 12501 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:11.137159 12501 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:11.137712 12501 heartbeater.cc:507] Master 127.11.111.60:36039 requested a full tablet report, sending...
I20251213 20:59:11.138630 11981 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:11.141254 12363 heartbeater.cc:499] Master 127.11.111.61:39203 was elected leader, sending a full tablet report...
I20251213 20:59:11.141292 12122 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:11.143322 12503 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:11.143441 12503 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:11.143832 12503 heartbeater.cc:507] Master 127.11.111.61:39203 requested a full tablet report, sending...
I20251213 20:59:11.144656 12052 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:11.145723 12052 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.11.111.2:38265
I20251213 20:59:11.363350 12233 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68c0fc24cab047dcaa4cae2dbd628626)
I20251213 20:59:11.363649 12233 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:11.364462 12233 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:11.365073 12062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:11.365106 12132 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6736d1c95f384b9098c089a20b76e9ae" is_pre_election: true
I20251213 20:59:11.365938 11969 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:11.366189 12238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 68c0fc24cab047dcaa4cae2dbd628626)
I20251213 20:59:11.366364 12233 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251213 20:59:11.366372 12238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:11.367143 12238 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:11.367609 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
I20251213 20:59:11.367749 12062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:11.367858 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 2.
I20251213 20:59:11.368887 12107 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters: 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:11.369225 12238 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251213 20:59:11.369336 12238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 2 FOLLOWER]: Starting leader election (detected failure of leader 68c0fc24cab047dcaa4cae2dbd628626)
I20251213 20:59:11.369449 12238 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 2 FOLLOWER]: Advancing to term 3
I20251213 20:59:11.371917 12238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:11.372611 12238 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 3 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:11.373090 12062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626"
I20251213 20:59:11.373127 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276"
I20251213 20:59:11.373363 11992 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 2 FOLLOWER]: Advancing to term 3
I20251213 20:59:11.375511 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 3.
I20251213 20:59:11.375916 12107 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters: 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:11.376240 12238 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 FOLLOWER]: Leader election won for term 3
I20251213 20:59:11.376663 12238 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 LEADER]: Becoming Leader. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Running, Role: LEADER
I20251213 20:59:11.377032 12238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:11.378257 12509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 3 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:11.378494 12509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:11.379004 12510 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:11.380529 12510 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:11.381366 12510 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:11.381475 12510 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:11.382660 12510 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:11.383371 12510 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Loaded TSK: 0
I20251213 20:59:11.384023 12510 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:11.384687 12510 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:11.385341 12510 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:11.462522 12511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:11.462870 12511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:11.463178 12132 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 LEADER]: Rejecting Update request from peer 68c0fc24cab047dcaa4cae2dbd628626 for earlier term 2. Current term is 3. Ops: []
I20251213 20:59:11.464221 12512 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251213 20:59:11.464531 12512 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Peer 6736d1c95f384b9098c089a20b76e9ae log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251213 20:59:11.464666 12511 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 2 LEADER]: Stepping down as leader of term 2
I20251213 20:59:11.464736 12511 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Running, Role: LEADER
I20251213 20:59:11.465015 12511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:11.465453 12511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 2 FOLLOWER]: Advancing to term 3
W20251213 20:59:11.475701 12508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:11.476033 12508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:11.476150 12508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:11.484438 12508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:11.485141 12508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.111.3
I20251213 20:59:11.495067 12508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.111.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.111.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.111.3
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:11.496742 12508 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:11.498529 12508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:11.507854 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:11.507880 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:11.508800 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:11.510226 12508 server_base.cc:1047] running on GCE node
I20251213 20:59:11.510852 12508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:11.511868 12508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:11.513085 12508 hybrid_clock.cc:648] HybridClock initialized: now 1765659551513026 us; error 47 us; skew 500 ppm
I20251213 20:59:11.513460 12508 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:11.516101 12508 webserver.cc:492] Webserver started at http://127.11.111.3:39215/ using document root <none> and password file <none>
I20251213 20:59:11.516857 12508 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:11.516952 12508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:11.517314 12508 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251213 20:59:11.519760 12508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data/instance:
uuid: "c5976941698547af85c98dff50a9d45b"
format_stamp: "Formatted at 2025-12-13 20:59:11 on dist-test-slave-p3t2"
I20251213 20:59:11.520509 12508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal/instance:
uuid: "c5976941698547af85c98dff50a9d45b"
format_stamp: "Formatted at 2025-12-13 20:59:11 on dist-test-slave-p3t2"
I20251213 20:59:11.525756 12508 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251213 20:59:11.529083 12528 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:11.530480 12508 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251213 20:59:11.530615 12508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal
uuid: "c5976941698547af85c98dff50a9d45b"
format_stamp: "Formatted at 2025-12-13 20:59:11 on dist-test-slave-p3t2"
I20251213 20:59:11.530777 12508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:11.553771 12508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:11.554733 12508 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:11.554968 12508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:11.555853 12508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251213 20:59:11.557554 12508 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251213 20:59:11.557641 12508 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:11.557722 12508 ts_tablet_manager.cc:616] Registered 0 tablets
I20251213 20:59:11.557772 12508 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:11.594398 12508 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.3:32819
I20251213 20:59:11.594450 12640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.3:32819 every 8 connection(s)
I20251213 20:59:11.596372 12508 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/data/info.pb
I20251213 20:59:11.601647 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 12508
I20251213 20:59:11.601855 11708 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal/instance
I20251213 20:59:11.618387 12643 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:11.618427 12642 heartbeater.cc:344] Connected to a master server at 127.11.111.62:33723
I20251213 20:59:11.618468 12641 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:11.618870 12643 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:11.618865 12641 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:11.618863 12642 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:11.619968 12643 heartbeater.cc:507] Master 127.11.111.61:39203 requested a full tablet report, sending...
I20251213 20:59:11.620002 12641 heartbeater.cc:507] Master 127.11.111.60:36039 requested a full tablet report, sending...
I20251213 20:59:11.620030 12642 heartbeater.cc:507] Master 127.11.111.62:33723 requested a full tablet report, sending...
I20251213 20:59:11.621632 12122 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:11.621745 12052 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:11.621830 11981 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:11.623382 12122 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.11.111.3:33143
I20251213 20:59:11.630887 11708 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251213 20:59:11.647960 12503 heartbeater.cc:499] Master 127.11.111.61:39203 was elected leader, sending a full tablet report...
I20251213 20:59:11.673105 12122 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_ebe9a0444b8a40c7ac9d635be30c28b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.673405 12120 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_e09d8c7259974a359a976428a2b351bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.672166 12121 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_4ece212e4b3b44968882ae1e8ca68422"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.674988 12118 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_e15c8fc8dfd143aaa4b8d039e22cf63d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.677134 12119 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_1368482eba894f4c8b68346432f132d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.678547 12117 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_b7f4c624bfff43c8a5177f45a23ff719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.679514 12115 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_12be436a89df4cfcac0b036dbab2d770"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.679697 12113 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_b027752248784b2f9fff90508e2b62b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.680099 12116 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_0dc454009abd48db8c91e0aa3c76eb4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.680651 12114 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39040:
name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251213 20:59:11.682216 12122 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ebe9a0444b8a40c7ac9d635be30c28b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.687115 12115 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_12be436a89df4cfcac0b036dbab2d770 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.687489 12120 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e09d8c7259974a359a976428a2b351bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.687145 12118 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.689404 12119 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1368482eba894f4c8b68346432f132d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.689781 12117 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b7f4c624bfff43c8a5177f45a23ff719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.690426 12113 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b027752248784b2f9fff90508e2b62b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.690541 12121 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4ece212e4b3b44968882ae1e8ca68422 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.691630 12116 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0dc454009abd48db8c91e0aa3c76eb4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251213 20:59:11.717803 11708 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskli_hB8/build/asan/bin/kudu pid 12089
I20251213 20:59:11.719152 12152 generic_service.cc:196] Checking for leaks (request via RPC)
I20251213 20:59:11.877712 11708 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskli_hB8/build/asan/bin/kudu with pid 12089
W20251213 20:59:11.906744 12650 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:11.916471 12652 negotiation.cc:337] Failed RPC negotiation. Trace:
1213 20:59:11.908294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.111.60:36039 (local address 127.0.0.1:39052)
1213 20:59:11.909060 (+ 766us) negotiation.cc:107] Waiting for socket to connect
1213 20:59:11.909081 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1213 20:59:11.909282 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1213 20:59:11.914888 (+ 5606us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":487}
I20251213 20:59:11.927417 12511 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:11.927631 12511 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:11.928328 12511 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:11.929607 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
I20251213 20:59:11.929845 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c0fc24cab047dcaa4cae2dbd628626 in term 3.
I20251213 20:59:11.930464 12037 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:11.930867 12511 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251213 20:59:11.931003 12511 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:11.931095 12511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 3 FOLLOWER]: Advancing to term 4
I20251213 20:59:11.934103 12511 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:11.935019 12511 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 4 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:11.935411 11992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276"
I20251213 20:59:11.935612 11992 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 3 FOLLOWER]: Advancing to term 4
I20251213 20:59:11.938280 11992 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c0fc24cab047dcaa4cae2dbd628626 in term 4.
I20251213 20:59:11.938793 12037 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:11.939127 12511 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 4 FOLLOWER]: Leader election won for term 4
I20251213 20:59:11.939284 12511 raft_consensus.cc:697] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 4 LEADER]: Becoming Leader. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Running, Role: LEADER
I20251213 20:59:11.939584 12511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
W20251213 20:59:11.947777 12038 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:11.948799 12512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c0fc24cab047dcaa4cae2dbd628626. Latest consensus state: current_term: 4 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:11.949079 12512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: LEADER
W20251213 20:59:11.949158 12038 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:11.949946 12514 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:11.952361 12514 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:11.953361 12514 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:11.953467 12514 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:11.954892 12514 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:11.955833 12514 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Loaded TSK: 0
I20251213 20:59:11.956349 12514 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:11.957296 12514 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:11.958209 12514 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:11.963158 12052 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_b027752248784b2f9fff90508e2b62b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.962131 12051 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.963642 12049 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_0dc454009abd48db8c91e0aa3c76eb4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.963922 12047 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_ebe9a0444b8a40c7ac9d635be30c28b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.963697 12045 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_1368482eba894f4c8b68346432f132d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.962761 12046 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_e09d8c7259974a359a976428a2b351bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.963613 12043 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_b7f4c624bfff43c8a5177f45a23ff719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.963478 12044 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_e15c8fc8dfd143aaa4b8d039e22cf63d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.962463 12048 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_4ece212e4b3b44968882ae1e8ca68422"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:11.961732 12050 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_12be436a89df4cfcac0b036dbab2d770"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251213 20:59:11.974215 12052 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b027752248784b2f9fff90508e2b62b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.976923 12045 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1368482eba894f4c8b68346432f132d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.977244 12049 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0dc454009abd48db8c91e0aa3c76eb4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.976955 12047 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ebe9a0444b8a40c7ac9d635be30c28b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.979036 12044 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.979946 12043 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b7f4c624bfff43c8a5177f45a23ff719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.980823 12046 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e09d8c7259974a359a976428a2b351bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.981775 12048 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4ece212e4b3b44968882ae1e8ca68422 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.982671 12051 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ceb9db759caa483dbbe2ac726dc7f8bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251213 20:59:11.983623 12050 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_12be436a89df4cfcac0b036dbab2d770 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251213 20:59:12.126061 12641 heartbeater.cc:499] Master 127.11.111.60:36039 was elected leader, sending a full tablet report...
W20251213 20:59:12.128242 12641 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:12.135326 12361 heartbeater.cc:499] Master 127.11.111.60:36039 was elected leader, sending a full tablet report...
W20251213 20:59:12.138406 12361 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:12.143370 12501 heartbeater.cc:499] Master 127.11.111.60:36039 was elected leader, sending a full tablet report...
W20251213 20:59:12.150957 12501 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:12.212373 12052 catalog_manager.cc:2174] created HMS catalog entry for table [id=abe1a66f87914e24b17972fa42e24765]
W20251213 20:59:12.216980 12038 consensus_peers.cc:597] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 -> Peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Couldn't send request to peer 6736d1c95f384b9098c089a20b76e9ae. Status: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251213 20:59:12.218386 11992 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 4 FOLLOWER]: Refusing update from remote peer 68c0fc24cab047dcaa4cae2dbd628626: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251213 20:59:12.219259 12511 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251213 20:59:12.245425 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c0fc24cab047dcaa4cae2dbd628626. Latest consensus state: current_term: 4 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:12.245713 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:12.247753 12694 mvcc.cc:204] Tried to move back new op lower bound from 7232141525870747648 to 7232141524746899456. Current Snapshot: MvccSnapshot[applied={T|T < 7232141525870747648}]
I20251213 20:59:12.251222 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:12.251436 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:12.254292 12696 mvcc.cc:204] Tried to move back new op lower bound from 7232141525870747648 to 7232141524746899456. Current Snapshot: MvccSnapshot[applied={T|T < 7232141525870747648}]
I20251213 20:59:12.255038 12512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:12.255287 12512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:12.258730 12045 catalog_manager.cc:2174] created HMS catalog entry for table [id=3defb3020c584a7587e3632511396bd3]
I20251213 20:59:12.272563 12052 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55444:
table { table_name: "default.Table_b027752248784b2f9fff90508e2b62b4" } new_table_name: "default.Table_31c612ce45a347a2b23e54b30cf823be" modify_external_catalogs: true
I20251213 20:59:12.292019 12045 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55444:
table { table_name: "default.Table_1368482eba894f4c8b68346432f132d8" } new_table_name: "default.Table_3c93b9a42ca143889e8ab38b6fc45d47" modify_external_catalogs: true
I20251213 20:59:12.299427 12049 catalog_manager.cc:2174] created HMS catalog entry for table [id=d2519a98a46e45618d21e90dd0348153]
I20251213 20:59:12.312371 12576 tablet_service.cc:1505] Processing CreateTablet for tablet b01ca5fbe5874c65adb18918adeae5e0 (DEFAULT_TABLE table=default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.315563 12576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b01ca5fbe5874c65adb18918adeae5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.339133 12708 tablet_bootstrap.cc:492] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.340566 12049 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55444:
table { table_name: "default.Table_0dc454009abd48db8c91e0aa3c76eb4c" } new_table_name: "default.Table_7c733924e7c54175b89a1b37e0095c4b" modify_external_catalogs: true
I20251213 20:59:12.345356 12708 tablet_bootstrap.cc:654] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.346855 12708 log.cc:826] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:12.347013 12436 tablet_service.cc:1505] Processing CreateTablet for tablet b01ca5fbe5874c65adb18918adeae5e0 (DEFAULT_TABLE table=default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.347870 12435 tablet_service.cc:1505] Processing CreateTablet for tablet cb1f2d25f95c49b4918383371cc12368 (DEFAULT_TABLE table=default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.348971 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1f2d25f95c49b4918383371cc12368. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.353124 12434 tablet_service.cc:1505] Processing CreateTablet for tablet 13f876d369744ac3955296d88a12b1e7 (DEFAULT_TABLE table=default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.353348 12436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b01ca5fbe5874c65adb18918adeae5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.353742 12434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f876d369744ac3955296d88a12b1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.356269 12708 tablet_bootstrap.cc:492] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.356570 12708 ts_tablet_manager.cc:1403] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.010s
I20251213 20:59:12.362672 12576 tablet_service.cc:1505] Processing CreateTablet for tablet cb1f2d25f95c49b4918383371cc12368 (DEFAULT_TABLE table=default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.363126 12576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1f2d25f95c49b4918383371cc12368. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.363451 12295 tablet_service.cc:1505] Processing CreateTablet for tablet cb1f2d25f95c49b4918383371cc12368 (DEFAULT_TABLE table=default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.364090 12294 tablet_service.cc:1505] Processing CreateTablet for tablet 13f876d369744ac3955296d88a12b1e7 (DEFAULT_TABLE table=default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.364835 12575 tablet_service.cc:1505] Processing CreateTablet for tablet 13f876d369744ac3955296d88a12b1e7 (DEFAULT_TABLE table=default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.365118 12295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb1f2d25f95c49b4918383371cc12368. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.365432 12575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f876d369744ac3955296d88a12b1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.373782 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13f876d369744ac3955296d88a12b1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.374157 12708 raft_consensus.cc:359] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.374591 12708 raft_consensus.cc:385] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.374688 12708 raft_consensus.cc:740] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.375531 12708 consensus_queue.cc:260] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.363435 12296 tablet_service.cc:1505] Processing CreateTablet for tablet b01ca5fbe5874c65adb18918adeae5e0 (DEFAULT_TABLE table=default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.382141 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b01ca5fbe5874c65adb18918adeae5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.383872 12708 ts_tablet_manager.cc:1434] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.027s user 0.013s sys 0.005s
I20251213 20:59:12.389816 12711 tablet_bootstrap.cc:492] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.390563 12643 heartbeater.cc:499] Master 127.11.111.61:39203 was elected leader, sending a full tablet report...
I20251213 20:59:12.391914 12708 tablet_bootstrap.cc:492] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.393746 12708 tablet_bootstrap.cc:654] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.394922 12711 tablet_bootstrap.cc:654] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.396368 12711 log.cc:826] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:12.400010 12712 tablet_bootstrap.cc:492] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.406867 12708 tablet_bootstrap.cc:492] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.407130 12708 ts_tablet_manager.cc:1403] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20251213 20:59:12.408257 12708 raft_consensus.cc:359] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.408577 12708 raft_consensus.cc:385] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.408705 12708 raft_consensus.cc:740] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.409065 12708 consensus_queue.cc:260] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.413096 12708 ts_tablet_manager.cc:1434] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251213 20:59:12.416319 12708 tablet_bootstrap.cc:492] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.418256 12708 tablet_bootstrap.cc:654] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.420368 12712 tablet_bootstrap.cc:654] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.422848 12711 tablet_bootstrap.cc:492] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.423238 12711 ts_tablet_manager.cc:1403] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.000s
I20251213 20:59:12.423233 12712 log.cc:826] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:12.435429 12711 raft_consensus.cc:359] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.435922 12711 raft_consensus.cc:385] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.436033 12711 raft_consensus.cc:740] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.436801 12712 tablet_bootstrap.cc:492] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.437898 12712 ts_tablet_manager.cc:1403] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20251213 20:59:12.446646 12708 tablet_bootstrap.cc:492] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.446532 12711 consensus_queue.cc:260] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.446830 12708 ts_tablet_manager.cc:1403] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.006s
I20251213 20:59:12.447613 12708 raft_consensus.cc:359] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.447796 12708 raft_consensus.cc:385] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.447875 12708 raft_consensus.cc:740] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.448101 12708 consensus_queue.cc:260] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.448558 12708 ts_tablet_manager.cc:1434] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251213 20:59:12.448619 12711 ts_tablet_manager.cc:1434] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.025s user 0.010s sys 0.000s
I20251213 20:59:12.450457 12363 heartbeater.cc:499] Master 127.11.111.61:39203 was elected leader, sending a full tablet report...
I20251213 20:59:12.451634 12711 tablet_bootstrap.cc:492] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.453444 12711 tablet_bootstrap.cc:654] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.445863 12712 raft_consensus.cc:359] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.453860 12712 raft_consensus.cc:385] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.454021 12712 raft_consensus.cc:740] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.454910 12712 consensus_queue.cc:260] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.460196 12711 tablet_bootstrap.cc:492] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.460376 12711 ts_tablet_manager.cc:1403] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251213 20:59:12.460978 12712 ts_tablet_manager.cc:1434] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.023s user 0.009s sys 0.004s
I20251213 20:59:12.461409 12711 raft_consensus.cc:359] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.461609 12711 raft_consensus.cc:385] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.461694 12711 raft_consensus.cc:740] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.463158 12711 consensus_queue.cc:260] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.463644 12711 ts_tablet_manager.cc:1434] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251213 20:59:12.464952 12503 heartbeater.cc:499] Master 127.11.111.61:39203 was elected leader, sending a full tablet report...
I20251213 20:59:12.465087 12712 tablet_bootstrap.cc:492] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.466262 12711 tablet_bootstrap.cc:492] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.467971 12711 tablet_bootstrap.cc:654] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.473111 12712 tablet_bootstrap.cc:654] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.478608 12712 tablet_bootstrap.cc:492] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.478785 12712 ts_tablet_manager.cc:1403] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20251213 20:59:12.479810 12712 raft_consensus.cc:359] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.480014 12712 raft_consensus.cc:385] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.480098 12712 raft_consensus.cc:740] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.480388 12712 consensus_queue.cc:260] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.484370 12712 ts_tablet_manager.cc:1434] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251213 20:59:12.484663 12711 tablet_bootstrap.cc:492] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.484867 12711 ts_tablet_manager.cc:1403] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20251213 20:59:12.485702 12711 raft_consensus.cc:359] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.485898 12711 raft_consensus.cc:385] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.485972 12712 tablet_bootstrap.cc:492] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.485993 12711 raft_consensus.cc:740] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.486248 12711 consensus_queue.cc:260] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.486703 12711 ts_tablet_manager.cc:1434] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251213 20:59:12.487643 12712 tablet_bootstrap.cc:654] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.495750 12712 tablet_bootstrap.cc:492] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.495951 12712 ts_tablet_manager.cc:1403] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20251213 20:59:12.497210 12712 raft_consensus.cc:359] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.497493 12712 raft_consensus.cc:385] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.497588 12712 raft_consensus.cc:740] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.497869 12712 consensus_queue.cc:260] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.500458 12712 ts_tablet_manager.cc:1434] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
20:59:12.512 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_ebe9a0444b8a40c7ac9d635be30c28b0 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20251213 20:59:12.514256 12047 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=a737a1b13a974bdab79ea3dcbd72c779]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_ebe9a0444b8a40c7ac9d635be30c28b0 already exists)
I20251213 20:59:12.524930 12047 master_service.cc:954] ReplaceTablet: received request to replace tablet b01ca5fbe5874c65adb18918adeae5e0 from {username='slave'} at 127.0.0.1:55444
I20251213 20:59:12.535979 12047 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Sending DeleteTablet for 3 replicas of tablet b01ca5fbe5874c65adb18918adeae5e0
I20251213 20:59:12.537358 12575 tablet_service.cc:1552] Processing DeleteTablet for tablet b01ca5fbe5874c65adb18918adeae5e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d2e8c74b4ca427ab675b574516c1db1) from {username='slave'} at 127.0.0.1:56532
I20251213 20:59:12.541240 12719 raft_consensus.cc:493] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:12.541572 12719 raft_consensus.cc:515] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.543856 12719 leader_election.cc:290] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5976941698547af85c98dff50a9d45b (127.11.111.3:32819), 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:12.554526 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b01ca5fbe5874c65adb18918adeae5e0" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38" is_pre_election: true
I20251213 20:59:12.555094 12316 raft_consensus.cc:2468] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b75487eefe54cbdb4140dea8ab8a499 in term 0.
I20251213 20:59:12.555861 12390 leader_election.cc:304] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters:
I20251213 20:59:12.556347 12719 raft_consensus.cc:2804] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:12.556520 12719 raft_consensus.cc:493] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:12.556629 12719 raft_consensus.cc:3060] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.563259 12719 raft_consensus.cc:515] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.563985 12044 catalog_manager.cc:2174] created HMS catalog entry for table [id=c299c8ffc6a7470691ac95ab24cd6261]
I20251213 20:59:12.564565 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b01ca5fbe5874c65adb18918adeae5e0" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38"
I20251213 20:59:12.564764 12316 raft_consensus.cc:3060] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.567659 12316 raft_consensus.cc:2468] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b75487eefe54cbdb4140dea8ab8a499 in term 1.
I20251213 20:59:12.571974 12390 leader_election.cc:304] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters:
I20251213 20:59:12.574361 12044 master_service.cc:954] ReplaceTablet: received request to replace tablet b01ca5fbe5874c65adb18918adeae5e0 from {username='slave'} at 127.0.0.1:55444
I20251213 20:59:12.574920 12044 master_service.cc:954] ReplaceTablet: received request to replace tablet 13f876d369744ac3955296d88a12b1e7 from {username='slave'} at 127.0.0.1:55444
W20251213 20:59:12.575827 12364 tablet.cc:2378] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251213 20:59:12.582149 12725 raft_consensus.cc:2804] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:12.584247 12044 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Sending DeleteTablet for 3 replicas of tablet 13f876d369744ac3955296d88a12b1e7
I20251213 20:59:12.584792 12044 catalog_manager.cc:6724] ReplaceTablet: tablet 13f876d369744ac3955296d88a12b1e7 deleted and replaced by tablet ff555a43366144fdb7f5c857d351ef3b
I20251213 20:59:12.585011 12044 master_service.cc:954] ReplaceTablet: received request to replace tablet 13f876d369744ac3955296d88a12b1e7 from {username='slave'} at 127.0.0.1:55444
I20251213 20:59:12.585762 12710 raft_consensus.cc:493] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:12.586023 12710 raft_consensus.cc:515] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.586248 12434 tablet_service.cc:1552] Processing DeleteTablet for tablet 13f876d369744ac3955296d88a12b1e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ff555a43366144fdb7f5c857d351ef3b) from {username='slave'} at 127.0.0.1:37712
I20251213 20:59:12.588227 12710 leader_election.cc:290] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:12.593180 12044 master_service.cc:954] ReplaceTablet: received request to replace tablet b01ca5fbe5874c65adb18918adeae5e0 from {username='slave'} at 127.0.0.1:55444
I20251213 20:59:12.593739 12576 tablet_service.cc:1552] Processing DeleteTablet for tablet 13f876d369744ac3955296d88a12b1e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ff555a43366144fdb7f5c857d351ef3b) from {username='slave'} at 127.0.0.1:56532
I20251213 20:59:12.597114 12296 tablet_service.cc:1552] Processing DeleteTablet for tablet 13f876d369744ac3955296d88a12b1e7 with delete_type TABLET_DATA_DELETED (replaced by tablet ff555a43366144fdb7f5c857d351ef3b) from {username='slave'} at 127.0.0.1:55128
I20251213 20:59:12.599854 12729 tablet_replica.cc:333] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: stopping tablet replica
I20251213 20:59:12.600018 12044 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_5009d0ba1a394917a7101259634328ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:12.601756 12729 raft_consensus.cc:2243] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251213 20:59:12.602319 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--webserver_interface=127.11.111.60
--webserver_port=36499
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251213 20:59:12.604246 12729 raft_consensus.cc:2272] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251213 20:59:12.604898 12725 raft_consensus.cc:697] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 LEADER]: Becoming Leader. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Running, Role: LEADER
W20251213 20:59:12.605166 12644 tablet.cc:2378] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251213 20:59:12.605706 12725 consensus_queue.cc:237] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } }
I20251213 20:59:12.606575 12435 tablet_service.cc:1505] Processing CreateTablet for tablet 529995cf7f2e4b15b5e026178d59a916 (DEFAULT_TABLE table=default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.606966 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529995cf7f2e4b15b5e026178d59a916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.608670 12727 tablet_replica.cc:333] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: stopping tablet replica
I20251213 20:59:12.608633 12576 tablet_service.cc:1505] Processing CreateTablet for tablet 529995cf7f2e4b15b5e026178d59a916 (DEFAULT_TABLE table=default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.609107 12727 raft_consensus.cc:2243] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251213 20:59:12.609197 12576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529995cf7f2e4b15b5e026178d59a916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.609448 12727 raft_consensus.cc:2272] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251213 20:59:12.609738 12436 tablet_service.cc:1505] Processing CreateTablet for tablet 7d2e8c74b4ca427ab675b574516c1db1 (DEFAULT_TABLE table=default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.609786 12574 tablet_service.cc:1505] Processing CreateTablet for tablet 7d2e8c74b4ca427ab675b574516c1db1 (DEFAULT_TABLE table=default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.610049 12436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2e8c74b4ca427ab675b574516c1db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.610303 12574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2e8c74b4ca427ab675b574516c1db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.606876 12296 tablet_service.cc:1505] Processing CreateTablet for tablet 529995cf7f2e4b15b5e026178d59a916 (DEFAULT_TABLE table=default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.610724 12296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 529995cf7f2e4b15b5e026178d59a916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.615399 12708 tablet_bootstrap.cc:492] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.616531 12711 tablet_bootstrap.cc:492] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.617035 12708 tablet_bootstrap.cc:654] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.618091 12711 tablet_bootstrap.cc:654] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.619051 12294 tablet_service.cc:1505] Processing CreateTablet for tablet 7d2e8c74b4ca427ab675b574516c1db1 (DEFAULT_TABLE table=default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.619411 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2e8c74b4ca427ab675b574516c1db1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251213 20:59:12.625484 12504 tablet.cc:2378] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251213 20:59:12.628067 12712 tablet_bootstrap.cc:492] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.630494 12712 tablet_bootstrap.cc:654] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.630884 12727 ts_tablet_manager.cc:1916] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251213 20:59:12.634953 12711 tablet_bootstrap.cc:492] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.635172 12711 ts_tablet_manager.cc:1403] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20251213 20:59:12.636054 12711 raft_consensus.cc:359] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.636303 12711 raft_consensus.cc:385] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.636430 12711 raft_consensus.cc:740] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.636736 12711 consensus_queue.cc:260] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.641191 12711 ts_tablet_manager.cc:1434] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251213 20:59:12.648797 12711 tablet_bootstrap.cc:492] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.650660 12711 tablet_bootstrap.cc:654] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.653774 12436 tablet_service.cc:1505] Processing CreateTablet for tablet ff555a43366144fdb7f5c857d351ef3b (DEFAULT_TABLE table=default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.654273 12436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff555a43366144fdb7f5c857d351ef3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.657541 12719 leader_election.cc:290] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 election: Requested vote from peers c5976941698547af85c98dff50a9d45b (127.11.111.3:32819), 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:12.661494 12294 tablet_service.cc:1505] Processing CreateTablet for tablet ff555a43366144fdb7f5c857d351ef3b (DEFAULT_TABLE table=default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.661985 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff555a43366144fdb7f5c857d351ef3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.665521 12727 ts_tablet_manager.cc:1929] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251213 20:59:12.669857 12047 catalog_manager.cc:6724] ReplaceTablet: tablet b01ca5fbe5874c65adb18918adeae5e0 deleted and replaced by tablet 7d2e8c74b4ca427ab675b574516c1db1
I20251213 20:59:12.671862 12296 tablet_service.cc:1552] Processing DeleteTablet for tablet b01ca5fbe5874c65adb18918adeae5e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d2e8c74b4ca427ab675b574516c1db1) from {username='slave'} at 127.0.0.1:55128
I20251213 20:59:12.672665 12047 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_9e21ceee05264206866e8b0d62a537c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:12.675740 12729 ts_tablet_manager.cc:1916] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251213 20:59:12.678601 12574 tablet_service.cc:1505] Processing CreateTablet for tablet ff555a43366144fdb7f5c857d351ef3b (DEFAULT_TABLE table=default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.680450 12574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff555a43366144fdb7f5c857d351ef3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.682371 12711 tablet_bootstrap.cc:492] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.682560 12711 ts_tablet_manager.cc:1403] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20251213 20:59:12.683372 12711 raft_consensus.cc:359] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.683605 12711 raft_consensus.cc:385] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.683681 12711 raft_consensus.cc:740] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.683955 12711 consensus_queue.cc:260] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.670670 12435 tablet_service.cc:1552] Processing DeleteTablet for tablet b01ca5fbe5874c65adb18918adeae5e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d2e8c74b4ca427ab675b574516c1db1) from {username='slave'} at 127.0.0.1:37712
I20251213 20:59:12.685501 12727 log.cc:1199] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Deleting WAL directory at /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal/wals/13f876d369744ac3955296d88a12b1e7
I20251213 20:59:12.686074 12727 ts_tablet_manager.cc:1950] T 13f876d369744ac3955296d88a12b1e7 P 1b75487eefe54cbdb4140dea8ab8a499: Deleting consensus metadata
I20251213 20:59:12.687695 12711 ts_tablet_manager.cc:1434] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251213 20:59:12.690766 12711 tablet_bootstrap.cc:492] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.691079 12727 tablet_replica.cc:333] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: stopping tablet replica
I20251213 20:59:12.691326 12727 raft_consensus.cc:2243] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 LEADER]: Raft consensus shutting down.
I20251213 20:59:12.691838 12727 pending_rounds.cc:70] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Trying to abort 1 pending ops.
I20251213 20:59:12.692415 12711 tablet_bootstrap.cc:654] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.692724 12727 pending_rounds.cc:77] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232141527561388032 op_type: NO_OP noop_request { }
I20251213 20:59:12.693346 12039 catalog_manager.cc:4985] TS 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023): tablet 13f876d369744ac3955296d88a12b1e7 (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]) successfully deleted
I20251213 20:59:12.697947 12712 tablet_bootstrap.cc:492] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.698114 12712 ts_tablet_manager.cc:1403] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.007s
I20251213 20:59:12.698165 12727 raft_consensus.cc:2889] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251213 20:59:12.698470 12727 raft_consensus.cc:2272] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251213 20:59:12.699062 12708 tablet_bootstrap.cc:492] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.699227 12708 ts_tablet_manager.cc:1403] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.006s
I20251213 20:59:12.699065 12712 raft_consensus.cc:359] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.699335 12712 raft_consensus.cc:385] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.699432 12712 raft_consensus.cc:740] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.699678 12712 consensus_queue.cc:260] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.699885 12727 ts_tablet_manager.cc:1916] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251213 20:59:12.700172 12712 ts_tablet_manager.cc:1434] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251213 20:59:12.700815 12712 tablet_bootstrap.cc:492] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.702698 12712 tablet_bootstrap.cc:654] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.702807 12711 tablet_bootstrap.cc:492] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.702960 12711 ts_tablet_manager.cc:1403] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251213 20:59:12.703691 12711 raft_consensus.cc:359] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.703874 12711 raft_consensus.cc:385] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.703964 12711 raft_consensus.cc:740] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.703958 12708 raft_consensus.cc:359] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.704154 12708 raft_consensus.cc:385] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.704264 12708 raft_consensus.cc:740] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.705538 12596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b01ca5fbe5874c65adb18918adeae5e0" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b" is_pre_election: true
I20251213 20:59:12.706030 12596 raft_consensus.cc:2468] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b75487eefe54cbdb4140dea8ab8a499 in term 0.
I20251213 20:59:12.706077 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b01ca5fbe5874c65adb18918adeae5e0" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b"
I20251213 20:59:12.706351 12595 raft_consensus.cc:3060] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.707953 12708 consensus_queue.cc:260] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.709306 12595 raft_consensus.cc:2468] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b75487eefe54cbdb4140dea8ab8a499 in term 1.
I20251213 20:59:12.704209 12711 consensus_queue.cc:260] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.718326 12711 ts_tablet_manager.cc:1434] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251213 20:59:12.722368 12729 ts_tablet_manager.cc:1929] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251213 20:59:12.722575 12729 log.cc:1199] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Deleting WAL directory at /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal/wals/13f876d369744ac3955296d88a12b1e7
I20251213 20:59:12.722621 12708 ts_tablet_manager.cc:1434] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
W20251213 20:59:12.722551 12532 leader_election.cc:343] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023): Not found: Tablet not found: 13f876d369744ac3955296d88a12b1e7
I20251213 20:59:12.723057 12729 ts_tablet_manager.cc:1950] T 13f876d369744ac3955296d88a12b1e7 P 97f3d76e88294646bccacef46bb93c38: Deleting consensus metadata
I20251213 20:59:12.724892 12729 tablet_replica.cc:333] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: stopping tablet replica
I20251213 20:59:12.725001 12037 catalog_manager.cc:4985] TS 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681): tablet 13f876d369744ac3955296d88a12b1e7 (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]) successfully deleted
I20251213 20:59:12.725088 12729 raft_consensus.cc:2243] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251213 20:59:12.725247 12729 raft_consensus.cc:2272] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251213 20:59:12.726348 12708 tablet_bootstrap.cc:492] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.728314 12708 tablet_bootstrap.cc:654] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.728405 12729 ts_tablet_manager.cc:1916] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251213 20:59:12.729141 12712 tablet_bootstrap.cc:492] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.729411 12712 ts_tablet_manager.cc:1403] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20251213 20:59:12.730273 12712 raft_consensus.cc:359] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.732003 12712 raft_consensus.cc:385] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.732139 12712 raft_consensus.cc:740] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.732474 12712 consensus_queue.cc:260] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.733096 12712 ts_tablet_manager.cc:1434] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20251213 20:59:12.733791 12530 leader_election.cc:343] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681): Not found: Tablet not found: 13f876d369744ac3955296d88a12b1e7
I20251213 20:59:12.733965 12727 ts_tablet_manager.cc:1929] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251213 20:59:12.734104 12727 log.cc:1199] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Deleting WAL directory at /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-1/wal/wals/b01ca5fbe5874c65adb18918adeae5e0
I20251213 20:59:12.734246 12712 tablet_bootstrap.cc:492] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.734009 12530 leader_election.cc:304] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5976941698547af85c98dff50a9d45b; no voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38
I20251213 20:59:12.734556 12727 ts_tablet_manager.cc:1950] T b01ca5fbe5874c65adb18918adeae5e0 P 1b75487eefe54cbdb4140dea8ab8a499: Deleting consensus metadata
I20251213 20:59:12.734619 12710 raft_consensus.cc:2749] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251213 20:59:12.735757 12039 catalog_manager.cc:4985] TS 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023): tablet b01ca5fbe5874c65adb18918adeae5e0 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]) successfully deleted
I20251213 20:59:12.736785 12043 catalog_manager.cc:2174] created HMS catalog entry for table [id=f78dc8504fd04c5e8dc48276891a7ba3]
I20251213 20:59:12.742292 12712 tablet_bootstrap.cc:654] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.743726 12729 ts_tablet_manager.cc:1929] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251213 20:59:12.743897 12729 log.cc:1199] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: Deleting WAL directory at /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-0/wal/wals/b01ca5fbe5874c65adb18918adeae5e0
I20251213 20:59:12.744238 12729 ts_tablet_manager.cc:1950] T b01ca5fbe5874c65adb18918adeae5e0 P 97f3d76e88294646bccacef46bb93c38: Deleting consensus metadata
I20251213 20:59:12.747087 12037 catalog_manager.cc:4985] TS 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681): tablet b01ca5fbe5874c65adb18918adeae5e0 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]) successfully deleted
I20251213 20:59:12.753558 12712 tablet_bootstrap.cc:492] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.753742 12712 ts_tablet_manager.cc:1403] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.002s
I20251213 20:59:12.754581 12712 raft_consensus.cc:359] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.754827 12712 raft_consensus.cc:385] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.754909 12712 raft_consensus.cc:740] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.755141 12712 consensus_queue.cc:260] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.755651 12712 ts_tablet_manager.cc:1434] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251213 20:59:12.769601 12294 tablet_service.cc:1505] Processing CreateTablet for tablet cd58b344007c431ab5d8771d201733b7 (DEFAULT_TABLE table=default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.770128 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd58b344007c431ab5d8771d201733b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.774771 12435 tablet_service.cc:1505] Processing CreateTablet for tablet cd58b344007c431ab5d8771d201733b7 (DEFAULT_TABLE table=default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.775238 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd58b344007c431ab5d8771d201733b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.778914 12711 tablet_bootstrap.cc:492] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.780664 12711 tablet_bootstrap.cc:654] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.784642 12043 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_e448c113874f43c6b7a38f4b918a08a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:12.785364 12712 tablet_bootstrap.cc:492] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.785328 12574 tablet_service.cc:1505] Processing CreateTablet for tablet cd58b344007c431ab5d8771d201733b7 (DEFAULT_TABLE table=default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.785864 12574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd58b344007c431ab5d8771d201733b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.787004 12712 tablet_bootstrap.cc:654] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.794258 12723 tablet_replica.cc:333] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: stopping tablet replica
I20251213 20:59:12.794876 12723 raft_consensus.cc:2243] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251213 20:59:12.795255 12708 tablet_bootstrap.cc:492] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.795430 12708 ts_tablet_manager.cc:1403] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.002s
I20251213 20:59:12.795468 12723 raft_consensus.cc:2272] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251213 20:59:12.796417 12708 raft_consensus.cc:359] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.796631 12708 raft_consensus.cc:385] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.796706 12708 raft_consensus.cc:740] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.796998 12708 consensus_queue.cc:260] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.799263 12708 ts_tablet_manager.cc:1434] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251213 20:59:12.799610 12708 tablet_bootstrap.cc:492] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.800015 12711 tablet_bootstrap.cc:492] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.800185 12711 ts_tablet_manager.cc:1403] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20251213 20:59:12.800944 12711 raft_consensus.cc:359] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.801127 12711 raft_consensus.cc:385] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.801185 12711 raft_consensus.cc:740] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.801275 12708 tablet_bootstrap.cc:654] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.801491 12711 consensus_queue.cc:260] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.803654 12723 ts_tablet_manager.cc:1916] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251213 20:59:12.805675 12711 ts_tablet_manager.cc:1434] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251213 20:59:12.805725 12712 tablet_bootstrap.cc:492] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.805912 12712 ts_tablet_manager.cc:1403] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.005s
I20251213 20:59:12.806762 12712 raft_consensus.cc:359] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.806959 12712 raft_consensus.cc:385] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.807035 12712 raft_consensus.cc:740] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.807307 12712 consensus_queue.cc:260] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.809635 12719 raft_consensus.cc:493] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:12.809782 12719 raft_consensus.cc:515] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.810374 12719 leader_election.cc:290] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.811086 12712 ts_tablet_manager.cc:1434] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251213 20:59:12.811055 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd58b344007c431ab5d8771d201733b7" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38" is_pre_election: true
I20251213 20:59:12.811285 12316 raft_consensus.cc:2468] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b75487eefe54cbdb4140dea8ab8a499 in term 0.
I20251213 20:59:12.811662 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd58b344007c431ab5d8771d201733b7" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b" is_pre_election: true
W20251213 20:59:12.812806 12391 leader_election.cc:343] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5976941698547af85c98dff50a9d45b (127.11.111.3:32819): Illegal state: must be running to vote when last-logged opid is not known
I20251213 20:59:12.813469 12390 leader_election.cc:304] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters: c5976941698547af85c98dff50a9d45b
I20251213 20:59:12.813956 12719 raft_consensus.cc:2804] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:12.814062 12719 raft_consensus.cc:493] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:12.814144 12719 raft_consensus.cc:3060] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.814240 12708 tablet_bootstrap.cc:492] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.814376 12708 ts_tablet_manager.cc:1403] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20251213 20:59:12.815140 12708 raft_consensus.cc:359] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.815302 12708 raft_consensus.cc:385] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.815374 12708 raft_consensus.cc:740] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.815605 12708 consensus_queue.cc:260] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.816267 12708 ts_tablet_manager.cc:1434] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251213 20:59:12.816788 12719 raft_consensus.cc:515] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.817494 12719 leader_election.cc:290] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 election: Requested vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.817655 12708 tablet_bootstrap.cc:492] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.818506 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd58b344007c431ab5d8771d201733b7" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b"
W20251213 20:59:12.819301 12391 leader_election.cc:343] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5976941698547af85c98dff50a9d45b (127.11.111.3:32819): Illegal state: must be running to vote when last-logged opid is not known
I20251213 20:59:12.820003 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd58b344007c431ab5d8771d201733b7" candidate_uuid: "1b75487eefe54cbdb4140dea8ab8a499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38"
I20251213 20:59:12.820207 12316 raft_consensus.cc:3060] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.820881 12723 ts_tablet_manager.cc:1929] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251213 20:59:12.821040 12723 log.cc:1199] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Deleting WAL directory at /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal/wals/b01ca5fbe5874c65adb18918adeae5e0
I20251213 20:59:12.822678 12708 tablet_bootstrap.cc:654] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.822779 12316 raft_consensus.cc:2468] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b75487eefe54cbdb4140dea8ab8a499 in term 1.
I20251213 20:59:12.823328 12390 leader_election.cc:304] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters: c5976941698547af85c98dff50a9d45b
I20251213 20:59:12.823626 12723 ts_tablet_manager.cc:1950] T b01ca5fbe5874c65adb18918adeae5e0 P c5976941698547af85c98dff50a9d45b: Deleting consensus metadata
I20251213 20:59:12.823814 12719 raft_consensus.cc:2804] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:12.824482 12719 raft_consensus.cc:697] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 LEADER]: Becoming Leader. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Running, Role: LEADER
I20251213 20:59:12.824805 12719 consensus_queue.cc:237] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.825655 12038 catalog_manager.cc:4985] TS c5976941698547af85c98dff50a9d45b (127.11.111.3:32819): tablet b01ca5fbe5874c65adb18918adeae5e0 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]) successfully deleted
I20251213 20:59:12.825788 12723 tablet_replica.cc:333] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: stopping tablet replica
I20251213 20:59:12.825985 12723 raft_consensus.cc:2243] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Raft consensus shutting down.
I20251213 20:59:12.826179 12723 raft_consensus.cc:2272] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Raft consensus is shut down!
I20251213 20:59:12.831707 12723 ts_tablet_manager.cc:1916] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251213 20:59:12.833587 12717 raft_consensus.cc:493] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:12.833814 12717 raft_consensus.cc:515] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.836010 12717 leader_election.cc:290] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.841652 12723 ts_tablet_manager.cc:1929] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251213 20:59:12.841804 12723 log.cc:1199] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Deleting WAL directory at /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/ts-2/wal/wals/13f876d369744ac3955296d88a12b1e7
I20251213 20:59:12.842177 12723 ts_tablet_manager.cc:1950] T 13f876d369744ac3955296d88a12b1e7 P c5976941698547af85c98dff50a9d45b: Deleting consensus metadata
I20251213 20:59:12.843919 12708 tablet_bootstrap.cc:492] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.844105 12708 ts_tablet_manager.cc:1403] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.005s
I20251213 20:59:12.845026 12708 raft_consensus.cc:359] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.845247 12708 raft_consensus.cc:385] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.845333 12708 raft_consensus.cc:740] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.845678 12708 consensus_queue.cc:260] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.845863 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d2e8c74b4ca427ab675b574516c1db1" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b" is_pre_election: true
I20251213 20:59:12.845956 12038 catalog_manager.cc:4985] TS c5976941698547af85c98dff50a9d45b (127.11.111.3:32819): tablet 13f876d369744ac3955296d88a12b1e7 (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]) successfully deleted
I20251213 20:59:12.846060 12595 raft_consensus.cc:2468] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:12.846359 12708 ts_tablet_manager.cc:1434] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251213 20:59:12.848860 12710 raft_consensus.cc:493] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:12.849004 12710 raft_consensus.cc:515] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.851228 12251 leader_election.cc:304] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:12.852003 12710 leader_election.cc:290] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:12.852639 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff555a43366144fdb7f5c857d351ef3b" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38" is_pre_election: true
I20251213 20:59:12.852854 12316 raft_consensus.cc:2468] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:12.853639 12530 leader_election.cc:304] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:12.853787 12717 raft_consensus.cc:2804] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:12.853905 12717 raft_consensus.cc:493] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:12.853998 12717 raft_consensus.cc:3060] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.854025 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff555a43366144fdb7f5c857d351ef3b" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:12.854404 12456 raft_consensus.cc:2468] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:12.855018 12710 raft_consensus.cc:2804] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:12.855132 12710 raft_consensus.cc:493] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:12.855213 12710 raft_consensus.cc:3060] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.857002 12717 raft_consensus.cc:515] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.857666 12717 leader_election.cc:290] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Requested vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.857864 12717 raft_consensus.cc:493] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:12.857987 12717 raft_consensus.cc:515] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.858050 12710 raft_consensus.cc:515] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.858525 12717 leader_election.cc:290] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.859467 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff555a43366144fdb7f5c857d351ef3b" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38"
I20251213 20:59:12.859634 12316 raft_consensus.cc:3060] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.860466 12710 leader_election.cc:290] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Requested vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:12.861343 12596 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "529995cf7f2e4b15b5e026178d59a916" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b" is_pre_election: true
I20251213 20:59:12.861552 12596 raft_consensus.cc:2468] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:12.861804 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d2e8c74b4ca427ab675b574516c1db1" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b"
I20251213 20:59:12.861953 12595 raft_consensus.cc:3060] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.862198 12316 raft_consensus.cc:2468] T ff555a43366144fdb7f5c857d351ef3b P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
I20251213 20:59:12.862615 12530 leader_election.cc:304] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:12.862897 12710 raft_consensus.cc:2804] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:12.864454 12455 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d2e8c74b4ca427ab675b574516c1db1" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:12.864689 12595 raft_consensus.cc:2468] T 7d2e8c74b4ca427ab675b574516c1db1 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:12.864795 12455 raft_consensus.cc:2468] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:12.864735 12454 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d2e8c74b4ca427ab675b574516c1db1" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:12.865033 12454 raft_consensus.cc:3060] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.865077 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff555a43366144fdb7f5c857d351ef3b" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:12.865222 12456 raft_consensus.cc:3060] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.865192 12453 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "529995cf7f2e4b15b5e026178d59a916" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:12.865518 12453 raft_consensus.cc:2468] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:12.866271 12252 leader_election.cc:304] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters:
I20251213 20:59:12.866597 12747 raft_consensus.cc:2804] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:12.866816 12747 raft_consensus.cc:493] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:12.866940 12747 raft_consensus.cc:3060] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.867035 12251 leader_election.cc:304] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:12.868043 12454 raft_consensus.cc:2468] T 7d2e8c74b4ca427ab675b574516c1db1 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:12.868958 12456 raft_consensus.cc:2468] T ff555a43366144fdb7f5c857d351ef3b P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
I20251213 20:59:12.869253 12717 raft_consensus.cc:2804] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:12.873948 12747 raft_consensus.cc:515] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.874975 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "529995cf7f2e4b15b5e026178d59a916" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:12.875182 12456 raft_consensus.cc:3060] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.878379 12456 raft_consensus.cc:2468] T 529995cf7f2e4b15b5e026178d59a916 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:12.878934 12252 leader_election.cc:304] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters:
I20251213 20:59:12.883610 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "529995cf7f2e4b15b5e026178d59a916" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b"
I20251213 20:59:12.883877 12595 raft_consensus.cc:3060] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.886969 12595 raft_consensus.cc:2468] T 529995cf7f2e4b15b5e026178d59a916 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:12.888118 12747 leader_election.cc:290] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Requested vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.890347 12747 raft_consensus.cc:2804] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:12.890796 12747 raft_consensus.cc:697] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [term 1 LEADER]: Becoming Leader. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Running, Role: LEADER
I20251213 20:59:12.891558 12747 consensus_queue.cc:237] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.896996 12046 catalog_manager.cc:2174] created HMS catalog entry for table [id=e09c4151ad45495896b7cc754c5834c5]
I20251213 20:59:12.907078 12717 raft_consensus.cc:697] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [term 1 LEADER]: Becoming Leader. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Running, Role: LEADER
I20251213 20:59:12.907483 12717 consensus_queue.cc:237] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.917668 12749 raft_consensus.cc:493] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:12.918131 12749 raft_consensus.cc:515] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.919616 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1f2d25f95c49b4918383371cc12368" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:12.919837 12456 raft_consensus.cc:2468] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:12.920415 12252 leader_election.cc:304] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters:
I20251213 20:59:12.920791 12750 raft_consensus.cc:2804] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:12.920900 12750 raft_consensus.cc:493] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:12.921002 12750 raft_consensus.cc:3060] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.920997 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1f2d25f95c49b4918383371cc12368" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b" is_pre_election: true
I20251213 20:59:12.921226 12595 raft_consensus.cc:2468] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:12.921734 12749 leader_election.cc:290] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.923374 12575 tablet_service.cc:1505] Processing CreateTablet for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (DEFAULT_TABLE table=default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.923677 12435 tablet_service.cc:1505] Processing CreateTablet for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (DEFAULT_TABLE table=default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.923801 12575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f43dfbead524dc6aa1cddfbafe6e639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.923918 12750 raft_consensus.cc:515] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.924054 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f43dfbead524dc6aa1cddfbafe6e639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.925125 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1f2d25f95c49b4918383371cc12368" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:12.925335 12456 raft_consensus.cc:3060] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.927983 12456 raft_consensus.cc:2468] T cb1f2d25f95c49b4918383371cc12368 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:12.928620 12252 leader_election.cc:304] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38; no voters:
I20251213 20:59:12.929589 12708 tablet_bootstrap.cc:492] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:12.929787 12595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb1f2d25f95c49b4918383371cc12368" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b"
I20251213 20:59:12.930011 12595 raft_consensus.cc:3060] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:12.930156 12294 tablet_service.cc:1505] Processing CreateTablet for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (DEFAULT_TABLE table=default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:12.930563 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f43dfbead524dc6aa1cddfbafe6e639. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:12.931433 12708 tablet_bootstrap.cc:654] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.932619 12595 raft_consensus.cc:2468] T cb1f2d25f95c49b4918383371cc12368 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:12.933581 12750 leader_election.cc:290] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Requested vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:12.933935 12750 raft_consensus.cc:2804] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:12.934131 12750 raft_consensus.cc:697] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [term 1 LEADER]: Becoming Leader. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Running, Role: LEADER
I20251213 20:59:12.934477 12750 consensus_queue.cc:237] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.937976 12046 catalog_manager.cc:5654] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2). New cstate: current_term: 1 leader_uuid: "1b75487eefe54cbdb4140dea8ab8a499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } health_report { overall_health: UNKNOWN } } }
I20251213 20:59:12.945035 12711 tablet_bootstrap.cc:492] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:12.946120 12708 tablet_bootstrap.cc:492] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:12.946282 12708 ts_tablet_manager.cc:1403] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.003s
I20251213 20:59:12.946815 12711 tablet_bootstrap.cc:654] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.947110 12708 raft_consensus.cc:359] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.947325 12708 raft_consensus.cc:385] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.947407 12708 raft_consensus.cc:740] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.947693 12708 consensus_queue.cc:260] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.948194 12708 ts_tablet_manager.cc:1434] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251213 20:59:12.953961 12712 tablet_bootstrap.cc:492] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:12.955513 12046 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_0667e63914504edaa41a24b9c799bfba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:12.955726 12712 tablet_bootstrap.cc:654] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:12.956058 12710 raft_consensus.cc:697] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [term 1 LEADER]: Becoming Leader. State: Replica: c5976941698547af85c98dff50a9d45b, State: Running, Role: LEADER
I20251213 20:59:12.956570 12710 consensus_queue.cc:237] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.961576 12711 tablet_bootstrap.cc:492] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:12.961797 12711 ts_tablet_manager.cc:1403] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20251213 20:59:12.962738 12711 raft_consensus.cc:359] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.962939 12711 raft_consensus.cc:385] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.963047 12711 raft_consensus.cc:740] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.963325 12711 consensus_queue.cc:260] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.968467 12711 ts_tablet_manager.cc:1434] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251213 20:59:12.980243 12712 tablet_bootstrap.cc:492] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:12.980436 12712 ts_tablet_manager.cc:1403] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.022s
I20251213 20:59:12.981285 12712 raft_consensus.cc:359] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.981529 12712 raft_consensus.cc:385] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:12.981634 12712 raft_consensus.cc:740] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:12.981902 12712 consensus_queue.cc:260] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:12.983337 12712 ts_tablet_manager.cc:1434] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251213 20:59:13.004115 12048 catalog_manager.cc:2174] created HMS catalog entry for table [id=62f666dd15494a0eb095b9f9ded82d18]
I20251213 20:59:13.010914 12048 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55444 (request call id 10) took 1049 ms. Trace:
I20251213 20:59:13.011098 12048 rpcz_store.cc:276] 1213 20:59:11.961333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1213 20:59:11.962233 (+ 900us) service_pool.cc:225] Handling call
1213 20:59:11.981882 (+ 19649us) catalog_manager.cc:2101] Acquired catalog manager lock
1213 20:59:11.982284 (+ 402us) catalog_manager.cc:2147] Created new table and tablet info
1213 20:59:13.004046 (+1021762us) catalog_manager.cc:2173] Created new table in HMS catalog
1213 20:59:13.010543 (+ 6497us) catalog_manager.cc:2207] Wrote table and tablets to system table
1213 20:59:13.010720 (+ 177us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1213 20:59:13.010860 (+ 140us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1213 20:59:13.005071 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1213 20:59:13.005230 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
1213 20:59:13.005239 (+ 9us) write_op.cc:435] Acquired schema lock
1213 20:59:13.005242 (+ 3us) tablet.cc:660] Decoding operations
1213 20:59:13.005348 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
1213 20:59:13.005364 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1213 20:59:13.005370 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1213 20:59:13.005482 (+ 112us) tablet.cc:699] Row locks acquired
1213 20:59:13.005485 (+ 3us) write_op.cc:260] PREPARE: finished
1213 20:59:13.005532 (+ 47us) write_op.cc:270] Start()
1213 20:59:13.005573 (+ 41us) write_op.cc:276] Timestamp: P: 1765659553005492 usec, L: 0
1213 20:59:13.005577 (+ 4us) op_driver.cc:348] REPLICATION: starting
1213 20:59:13.005798 (+ 221us) log.cc:844] Serialized 432 byte log entry
1213 20:59:13.009844 (+ 4046us) op_driver.cc:464] REPLICATION: finished
1213 20:59:13.009979 (+ 135us) write_op.cc:301] APPLY: starting
1213 20:59:13.009997 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1213 20:59:13.010051 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1213 20:59:13.010100 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
1213 20:59:13.010225 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
1213 20:59:13.010264 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1213 20:59:13.010271 (+ 7us) write_op.cc:312] APPLY: finished
1213 20:59:13.010355 (+ 84us) log.cc:844] Serialized 28 byte log entry
1213 20:59:13.010419 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1213 20:59:13.010441 (+ 22us) write_op.cc:454] Released schema lock
1213 20:59:13.010448 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1213 20:59:13.010526 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1213 20:59:13.010530 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":914287,"Hive Metastore.run_cpu_time_us":307,"Hive Metastore.run_wall_time_us":107041,"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":536,"apply.run_wall_time_us":535,"num_ops":2,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":859,"prepare.run_wall_time_us":860,"raft.queue_time_us":988,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":282,"replication_time_us":4235}]]}
I20251213 20:59:13.020566 12048 catalog_manager.cc:5654] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b reported cstate change: term changed from 0 to 1, leader changed from <none> to c5976941698547af85c98dff50a9d45b (127.11.111.3). New cstate: current_term: 1 leader_uuid: "c5976941698547af85c98dff50a9d45b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } health_report { overall_health: HEALTHY } } }
I20251213 20:59:13.021862 12435 tablet_service.cc:1505] Processing CreateTablet for tablet fa028486f240481f9081ea514153e384 (DEFAULT_TABLE table=default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.021859 12575 tablet_service.cc:1505] Processing CreateTablet for tablet fa028486f240481f9081ea514153e384 (DEFAULT_TABLE table=default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.022308 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa028486f240481f9081ea514153e384. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.022308 12575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa028486f240481f9081ea514153e384. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.026248 12294 tablet_service.cc:1505] Processing CreateTablet for tablet fa028486f240481f9081ea514153e384 (DEFAULT_TABLE table=default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.026700 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa028486f240481f9081ea514153e384. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.028077 12708 tablet_bootstrap.cc:492] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:13.029750 12708 tablet_bootstrap.cc:654] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.032714 12048 catalog_manager.cc:5654] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97f3d76e88294646bccacef46bb93c38 (127.11.111.1). New cstate: current_term: 1 leader_uuid: "97f3d76e88294646bccacef46bb93c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } health_report { overall_health: UNKNOWN } } }
I20251213 20:59:13.033313 12048 catalog_manager.cc:5654] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97f3d76e88294646bccacef46bb93c38 (127.11.111.1). New cstate: current_term: 1 leader_uuid: "97f3d76e88294646bccacef46bb93c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } health_report { overall_health: UNKNOWN } } }
I20251213 20:59:13.034978 12048 catalog_manager.cc:5654] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97f3d76e88294646bccacef46bb93c38 (127.11.111.1). New cstate: current_term: 1 leader_uuid: "97f3d76e88294646bccacef46bb93c38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } health_report { overall_health: UNKNOWN } } }
I20251213 20:59:13.035696 12712 tablet_bootstrap.cc:492] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:13.037555 12712 tablet_bootstrap.cc:654] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.044854 12708 tablet_bootstrap.cc:492] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:13.045068 12708 ts_tablet_manager.cc:1403] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20251213 20:59:13.046111 12711 tablet_bootstrap.cc:492] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:13.046047 12708 raft_consensus.cc:359] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.046273 12708 raft_consensus.cc:385] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.046341 12708 raft_consensus.cc:740] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.046684 12708 consensus_queue.cc:260] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.047277 12708 ts_tablet_manager.cc:1434] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251213 20:59:13.048028 12711 tablet_bootstrap.cc:654] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.052628 12048 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55444:
table { table_name: "default.Table_e15c8fc8dfd143aaa4b8d039e22cf63d" } new_table_name: "default.Table_40016a174bfc4e3c9f29ec786589b626" modify_external_catalogs: true
I20251213 20:59:13.053900 12712 tablet_bootstrap.cc:492] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:13.054088 12712 ts_tablet_manager.cc:1403] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20251213 20:59:13.054860 12712 raft_consensus.cc:359] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.055076 12712 raft_consensus.cc:385] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.055152 12712 raft_consensus.cc:740] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.055400 12712 consensus_queue.cc:260] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.056244 12712 ts_tablet_manager.cc:1434] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251213 20:59:13.066627 12051 catalog_manager.cc:2174] created HMS catalog entry for table [id=eec24cb8c3c142f48350b073c99e4d60]
I20251213 20:59:13.073179 12051 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55444 (request call id 7) took 1112 ms. Trace:
I20251213 20:59:13.073359 12051 rpcz_store.cc:276] 1213 20:59:11.960899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1213 20:59:11.961886 (+ 987us) service_pool.cc:225] Handling call
1213 20:59:11.982787 (+ 20901us) catalog_manager.cc:2101] Acquired catalog manager lock
1213 20:59:11.983194 (+ 407us) catalog_manager.cc:2147] Created new table and tablet info
1213 20:59:13.066554 (+1083360us) catalog_manager.cc:2173] Created new table in HMS catalog
1213 20:59:13.072847 (+ 6293us) catalog_manager.cc:2207] Wrote table and tablets to system table
1213 20:59:13.072983 (+ 136us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1213 20:59:13.073124 (+ 141us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1213 20:59:13.067573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1213 20:59:13.067707 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
1213 20:59:13.067716 (+ 9us) write_op.cc:435] Acquired schema lock
1213 20:59:13.067718 (+ 2us) tablet.cc:660] Decoding operations
1213 20:59:13.067827 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
1213 20:59:13.067843 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1213 20:59:13.067849 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1213 20:59:13.067960 (+ 111us) tablet.cc:699] Row locks acquired
1213 20:59:13.067963 (+ 3us) write_op.cc:260] PREPARE: finished
1213 20:59:13.068009 (+ 46us) write_op.cc:270] Start()
1213 20:59:13.068049 (+ 40us) write_op.cc:276] Timestamp: P: 1765659553067964 usec, L: 0
1213 20:59:13.068053 (+ 4us) op_driver.cc:348] REPLICATION: starting
1213 20:59:13.068255 (+ 202us) log.cc:844] Serialized 432 byte log entry
1213 20:59:13.071958 (+ 3703us) op_driver.cc:464] REPLICATION: finished
1213 20:59:13.072081 (+ 123us) write_op.cc:301] APPLY: starting
1213 20:59:13.072096 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1213 20:59:13.072144 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1213 20:59:13.072179 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1213 20:59:13.072282 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
1213 20:59:13.072318 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1213 20:59:13.072325 (+ 7us) write_op.cc:312] APPLY: finished
1213 20:59:13.072406 (+ 81us) log.cc:844] Serialized 28 byte log entry
1213 20:59:13.072482 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1213 20:59:13.072504 (+ 22us) write_op.cc:454] Released schema lock
1213 20:59:13.072511 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1213 20:59:13.072580 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1213 20:59:13.072584 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1020503,"Hive Metastore.run_cpu_time_us":295,"Hive Metastore.run_wall_time_us":62447,"mutex_wait_us":1516,"child_traces":[["op",{"apply.queue_time_us":78,"apply.run_cpu_time_us":490,"apply.run_wall_time_us":489,"num_ops":2,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":805,"prepare.run_wall_time_us":805,"raft.queue_time_us":958,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":289,"replication_time_us":3875}]]}
I20251213 20:59:13.075246 12710 raft_consensus.cc:493] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:13.075446 12710 raft_consensus.cc:515] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.076144 12710 leader_election.cc:290] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:13.077064 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f43dfbead524dc6aa1cddfbafe6e639" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38" is_pre_election: true
I20251213 20:59:13.077340 12316 raft_consensus.cc:2468] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:13.077921 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f43dfbead524dc6aa1cddfbafe6e639" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:13.078153 12456 raft_consensus.cc:2468] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:13.078640 12532 leader_election.cc:304] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:13.079262 12710 raft_consensus.cc:2804] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:13.079360 12710 raft_consensus.cc:493] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:13.079461 12710 raft_consensus.cc:3060] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.082111 12710 raft_consensus.cc:515] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.083083 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f43dfbead524dc6aa1cddfbafe6e639" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38"
I20251213 20:59:13.083292 12316 raft_consensus.cc:3060] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.086352 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f43dfbead524dc6aa1cddfbafe6e639" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:13.086578 12456 raft_consensus.cc:3060] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.089190 12456 raft_consensus.cc:2468] T 7f43dfbead524dc6aa1cddfbafe6e639 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
I20251213 20:59:13.090206 12532 leader_election.cc:304] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:13.090581 12748 raft_consensus.cc:2804] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:13.091810 12316 raft_consensus.cc:2468] T 7f43dfbead524dc6aa1cddfbafe6e639 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
I20251213 20:59:13.101904 12435 tablet_service.cc:1505] Processing CreateTablet for tablet da4c8619ce49439cb7b84eb17d587fb4 (DEFAULT_TABLE table=default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.102362 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4c8619ce49439cb7b84eb17d587fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.106154 12575 tablet_service.cc:1505] Processing CreateTablet for tablet da4c8619ce49439cb7b84eb17d587fb4 (DEFAULT_TABLE table=default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.106444 12710 leader_election.cc:290] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Requested vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:13.106670 12575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4c8619ce49439cb7b84eb17d587fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.108960 12051 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55444:
table { table_name: "default.Table_b7f4c624bfff43c8a5177f45a23ff719" } new_table_name: "default.Table_cbfd66dd739f453f8b64a7f6eaf0b21b" modify_external_catalogs: true
I20251213 20:59:13.117635 12712 tablet_bootstrap.cc:492] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
I20251213 20:59:13.119263 12712 tablet_bootstrap.cc:654] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.119446 12294 tablet_service.cc:1505] Processing CreateTablet for tablet da4c8619ce49439cb7b84eb17d587fb4 (DEFAULT_TABLE table=default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.120275 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da4c8619ce49439cb7b84eb17d587fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.121660 12708 tablet_bootstrap.cc:492] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:13.123217 12708 tablet_bootstrap.cc:654] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.124150 12050 catalog_manager.cc:2174] created HMS catalog entry for table [id=e211037a5d9045ada23784889d3fdb72]
I20251213 20:59:13.124567 12748 raft_consensus.cc:697] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [term 1 LEADER]: Becoming Leader. State: Replica: c5976941698547af85c98dff50a9d45b, State: Running, Role: LEADER
I20251213 20:59:13.124900 12748 consensus_queue.cc:237] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.126808 12711 tablet_bootstrap.cc:492] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:13.126981 12711 ts_tablet_manager.cc:1403] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.008s
I20251213 20:59:13.130568 12712 tablet_bootstrap.cc:492] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:13.130708 12712 ts_tablet_manager.cc:1403] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20251213 20:59:13.131233 12050 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:55444 (request call id 8) took 1170 ms. Trace:
I20251213 20:59:13.131374 12050 rpcz_store.cc:276] 1213 20:59:11.961088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1213 20:59:11.961153 (+ 65us) service_pool.cc:225] Handling call
1213 20:59:11.983763 (+ 22610us) catalog_manager.cc:2101] Acquired catalog manager lock
1213 20:59:11.984162 (+ 399us) catalog_manager.cc:2147] Created new table and tablet info
1213 20:59:13.124096 (+1139934us) catalog_manager.cc:2173] Created new table in HMS catalog
1213 20:59:13.130947 (+ 6851us) catalog_manager.cc:2207] Wrote table and tablets to system table
1213 20:59:13.131064 (+ 117us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1213 20:59:13.131181 (+ 117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1213 20:59:13.124950 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1213 20:59:13.125085 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1213 20:59:13.125093 (+ 8us) write_op.cc:435] Acquired schema lock
1213 20:59:13.125095 (+ 2us) tablet.cc:660] Decoding operations
1213 20:59:13.125189 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
1213 20:59:13.125204 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1213 20:59:13.125209 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1213 20:59:13.125293 (+ 84us) tablet.cc:699] Row locks acquired
1213 20:59:13.125295 (+ 2us) write_op.cc:260] PREPARE: finished
1213 20:59:13.125334 (+ 39us) write_op.cc:270] Start()
1213 20:59:13.125368 (+ 34us) write_op.cc:276] Timestamp: P: 1765659553125282 usec, L: 0
1213 20:59:13.125372 (+ 4us) op_driver.cc:348] REPLICATION: starting
1213 20:59:13.125637 (+ 265us) log.cc:844] Serialized 432 byte log entry
1213 20:59:13.129888 (+ 4251us) op_driver.cc:464] REPLICATION: finished
1213 20:59:13.130042 (+ 154us) write_op.cc:301] APPLY: starting
1213 20:59:13.130059 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1213 20:59:13.130109 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1213 20:59:13.130144 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1213 20:59:13.130259 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
1213 20:59:13.130297 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1213 20:59:13.130304 (+ 7us) write_op.cc:312] APPLY: finished
1213 20:59:13.130388 (+ 84us) log.cc:844] Serialized 28 byte log entry
1213 20:59:13.130452 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1213 20:59:13.130486 (+ 34us) write_op.cc:454] Released schema lock
1213 20:59:13.130493 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1213 20:59:13.130561 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1213 20:59:13.130565 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1082059,"Hive Metastore.run_cpu_time_us":289,"Hive Metastore.run_wall_time_us":57593,"child_traces":[["op",{"apply.queue_time_us":104,"apply.run_cpu_time_us":510,"apply.run_wall_time_us":510,"num_ops":2,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":803,"prepare.run_wall_time_us":804,"raft.queue_time_us":1166,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":267,"replication_time_us":4485}]]}
I20251213 20:59:13.132316 12050 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55444:
name: "default.Table_eb562e37a81541048c6a9757feca4265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:13.132825 12708 tablet_bootstrap.cc:492] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:13.132966 12708 ts_tablet_manager.cc:1403] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20251213 20:59:13.133872 12708 raft_consensus.cc:359] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.134047 12708 raft_consensus.cc:385] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.134126 12708 raft_consensus.cc:740] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.134339 12708 consensus_queue.cc:260] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.134841 12708 ts_tablet_manager.cc:1434] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251213 20:59:13.135489 12712 raft_consensus.cc:359] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.135665 12712 raft_consensus.cc:385] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.135736 12712 raft_consensus.cc:740] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.135931 12712 consensus_queue.cc:260] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.136719 12712 ts_tablet_manager.cc:1434] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251213 20:59:13.127725 12711 raft_consensus.cc:359] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.137037 12711 raft_consensus.cc:385] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.137115 12711 raft_consensus.cc:740] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.137357 12711 consensus_queue.cc:260] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.137876 12711 ts_tablet_manager.cc:1434] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251213 20:59:13.149734 12435 tablet_service.cc:1505] Processing CreateTablet for tablet f61276e680a24502bd0d0547a55b2abd (DEFAULT_TABLE table=default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.150298 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61276e680a24502bd0d0547a55b2abd. 1 dirs total, 0 dirs full, 0 dirs failed
F20251213 20:59:13.150344 12225 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_ebe9a0444b8a40c7ac9d635be30c28b0, message={"server":"","servicePrincipal":"","db":"default","table":"table_ebe9a0444b8a40c7ac9d635be30c28b0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ebe9a0444b8a40c7ac9d635be30c28b0\",\"kudu.table_id\":\"2d6fd96071954bcb92508900e2a386b6\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_b027752248784b2f9fff90508e2b62b4, message={"server":"","servicePrincipal":"","db":"default","table":"table_b027752248784b2f9fff90508e2b62b4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b027752248784b2f9fff90508e2b62b4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_b027752248784b2f9fff90508e2b62b4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b027752248784b2f9fff90508e2b62b4\",\"kudu.table_id\":\"abe1a66f87914e24b17972fa42e24765\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f996ac4fdcd google::LogMessage::Fail() at ??:0
I20251213 20:59:13.154170 12575 tablet_service.cc:1505] Processing CreateTablet for tablet f61276e680a24502bd0d0547a55b2abd (DEFAULT_TABLE table=default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.154680 12575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61276e680a24502bd0d0547a55b2abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.157357 12712 tablet_bootstrap.cc:492] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499: Bootstrap starting.
@ 0x7f996ac53b93 google::LogMessage::SendToLog() at ??:0
I20251213 20:59:13.159307 12712 tablet_bootstrap.cc:654] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.159879 12711 tablet_bootstrap.cc:492] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:13.161486 12711 tablet_bootstrap.cc:654] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
@ 0x7f996ac4f7cc google::LogMessage::Flush() at ??:0
@ 0x7f996ac4fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9981635aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20251213 20:59:13.181975 12294 tablet_service.cc:1505] Processing CreateTablet for tablet f61276e680a24502bd0d0547a55b2abd (DEFAULT_TABLE table=default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20251213 20:59:13.182523 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f61276e680a24502bd0d0547a55b2abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251213 20:59:13.185668 12708 tablet_bootstrap.cc:492] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b: Bootstrap starting.
I20251213 20:59:13.187389 12708 tablet_bootstrap.cc:654] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.189484 12708 tablet_bootstrap.cc:492] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b: No bootstrap required, opened a new log
I20251213 20:59:13.189638 12708 ts_tablet_manager.cc:1403] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20251213 20:59:13.190402 12708 raft_consensus.cc:359] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.190605 12708 raft_consensus.cc:385] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.190668 12708 raft_consensus.cc:740] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5976941698547af85c98dff50a9d45b, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.191542 12708 consensus_queue.cc:260] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.192205 12708 ts_tablet_manager.cc:1434] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251213 20:59:13.198802 12711 tablet_bootstrap.cc:492] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:13.198947 12711 ts_tablet_manager.cc:1403] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.009s
I20251213 20:59:13.199748 12711 raft_consensus.cc:359] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.199908 12711 raft_consensus.cc:385] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.199981 12711 raft_consensus.cc:740] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.200192 12711 consensus_queue.cc:260] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.200675 12711 ts_tablet_manager.cc:1434] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251213 20:59:13.205737 12748 raft_consensus.cc:493] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:13.205933 12748 raft_consensus.cc:515] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.206621 12748 leader_election.cc:290] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
@ 0x7f9981634351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20251213 20:59:13.207295 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4c8619ce49439cb7b84eb17d587fb4" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38" is_pre_election: true
I20251213 20:59:13.207481 12316 raft_consensus.cc:2468] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:13.208066 12530 leader_election.cc:304] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:13.208180 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4c8619ce49439cb7b84eb17d587fb4" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:13.208370 12748 raft_consensus.cc:2804] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:13.208377 12456 raft_consensus.cc:2468] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:13.208452 12748 raft_consensus.cc:493] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:13.208539 12748 raft_consensus.cc:3060] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.211347 12748 raft_consensus.cc:515] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
@ 0x7f998163bfcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20251213 20:59:13.212340 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4c8619ce49439cb7b84eb17d587fb4" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38"
I20251213 20:59:13.212500 12316 raft_consensus.cc:3060] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.215188 12316 raft_consensus.cc:2468] T da4c8619ce49439cb7b84eb17d587fb4 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
I20251213 20:59:13.215852 12530 leader_election.cc:304] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:13.216154 12761 raft_consensus.cc:2804] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:13.216285 12761 raft_consensus.cc:697] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [term 1 LEADER]: Becoming Leader. State: Replica: c5976941698547af85c98dff50a9d45b, State: Running, Role: LEADER
I20251213 20:59:13.216547 12761 consensus_queue.cc:237] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
@ 0x7f998163be49 std::_Function_handler<>::_M_invoke() at ??:0
I20251213 20:59:13.217999 12748 leader_election.cc:290] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Requested vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:13.219257 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da4c8619ce49439cb7b84eb17d587fb4" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:13.219496 12456 raft_consensus.cc:3060] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.222800 12456 raft_consensus.cc:2468] T da4c8619ce49439cb7b84eb17d587fb4 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
I20251213 20:59:13.223764 12711 tablet_bootstrap.cc:492] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38: Bootstrap starting.
I20251213 20:59:13.225788 12711 tablet_bootstrap.cc:654] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38: Neither blocks nor log segments found. Creating new log.
I20251213 20:59:13.230993 12712 tablet_bootstrap.cc:492] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499: No bootstrap required, opened a new log
I20251213 20:59:13.231164 12712 ts_tablet_manager.cc:1403] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499: Time spent bootstrapping tablet: real 0.074s user 0.022s sys 0.018s
I20251213 20:59:13.232007 12712 raft_consensus.cc:359] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.232219 12712 raft_consensus.cc:385] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
@ 0xa8970d std::function<>::operator()() at ??:?
I20251213 20:59:13.232287 12712 raft_consensus.cc:740] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b75487eefe54cbdb4140dea8ab8a499, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.232551 12712 consensus_queue.cc:260] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.233103 12712 ts_tablet_manager.cc:1434] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251213 20:59:13.247916 12711 tablet_bootstrap.cc:492] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38: No bootstrap required, opened a new log
I20251213 20:59:13.248345 12748 raft_consensus.cc:493] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:13.248566 12748 raft_consensus.cc:515] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.249305 12748 leader_election.cc:290] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:13.249877 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa028486f240481f9081ea514153e384" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38" is_pre_election: true
I20251213 20:59:13.249881 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa028486f240481f9081ea514153e384" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:13.250113 12456 raft_consensus.cc:2468] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:13.250667 12532 leader_election.cc:304] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b75487eefe54cbdb4140dea8ab8a499, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:13.250962 12710 raft_consensus.cc:2804] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:13.250986 12711 ts_tablet_manager.cc:1403] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20251213 20:59:13.251065 12710 raft_consensus.cc:493] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:13.251142 12710 raft_consensus.cc:3060] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.251866 12711 raft_consensus.cc:359] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.252045 12711 raft_consensus.cc:385] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251213 20:59:13.252107 12711 raft_consensus.cc:740] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.252342 12711 consensus_queue.cc:260] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.253459 12316 raft_consensus.cc:2468] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 0.
I20251213 20:59:13.253829 12711 ts_tablet_manager.cc:1434] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251213 20:59:13.254776 12710 raft_consensus.cc:515] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
I20251213 20:59:13.255748 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa028486f240481f9081ea514153e384" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38"
@ 0x7f996d9c43f9 kudu::Thread::SuperviseThread() at ??:0
I20251213 20:59:13.255931 12316 raft_consensus.cc:3060] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
@ 0x7f9970fcf6db start_thread at ??:0
I20251213 20:59:13.258630 12316 raft_consensus.cc:2468] T fa028486f240481f9081ea514153e384 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
@ 0x7f9969b0071f clone at ??:0
I20251213 20:59:13.260409 12530 leader_election.cc:304] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:13.260687 12748 raft_consensus.cc:2804] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:13.261716 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa028486f240481f9081ea514153e384" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:13.261953 12456 raft_consensus.cc:3060] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.264947 12456 raft_consensus.cc:2468] T fa028486f240481f9081ea514153e384 P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5976941698547af85c98dff50a9d45b in term 1.
I20251213 20:59:13.266247 12710 leader_election.cc:290] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 election: Requested vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:13.268764 12748 raft_consensus.cc:697] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [term 1 LEADER]: Becoming Leader. State: Replica: c5976941698547af85c98dff50a9d45b, State: Running, Role: LEADER
I20251213 20:59:13.269070 12748 consensus_queue.cc:237] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } } peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } }
W20251213 20:59:13.459013 12733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:13.459340 12733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:13.459442 12733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:13.471843 12733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:13.471971 12733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:13.472054 12733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:13.472111 12733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:13.487912 12733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.111.60
--webserver_port=36499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:13.490134 12733 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20251213 20:59:13.491518 12651 connection.cc:537] client connection to 127.11.111.61:39203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:13.492219 12651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.111.61:39203)
W20251213 20:59:13.492729 12651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.111.61:39203)
W20251213 20:59:13.493440 12651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.111.61:39203)
I20251213 20:59:13.494103 12733 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:13.494153 12651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.111.61:39203)
W20251213 20:59:13.504917 12791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:13.505141 12790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:13.505661 12793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:13.506731 12733 server_base.cc:1047] running on GCE node
I20251213 20:59:13.507730 12733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:13.509214 12733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:13.510416 12733 hybrid_clock.cc:648] HybridClock initialized: now 1765659553510358 us; error 46 us; skew 500 ppm
I20251213 20:59:13.510749 12733 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:13.512889 12733 webserver.cc:492] Webserver started at http://127.11.111.60:36499/ using document root <none> and password file <none>
I20251213 20:59:13.513861 12733 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:13.513995 12733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:13.519382 12733 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20251213 20:59:13.522178 12799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:13.523779 12733 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251213 20:59:13.523958 12733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
uuid: "6736d1c95f384b9098c089a20b76e9ae"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:13.524626 12733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:13.543561 12733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:13.544394 12733 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:13.544663 12733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:13.549230 12748 raft_consensus.cc:493] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:13.549446 12748 raft_consensus.cc:515] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:13.550037 12748 leader_election.cc:290] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681), 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:13.550679 12316 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd58b344007c431ab5d8771d201733b7" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f3d76e88294646bccacef46bb93c38" is_pre_election: true
I20251213 20:59:13.550702 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd58b344007c431ab5d8771d201733b7" candidate_uuid: "c5976941698547af85c98dff50a9d45b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:13.550987 12316 raft_consensus.cc:2393] T cd58b344007c431ab5d8771d201733b7 P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5976941698547af85c98dff50a9d45b in current term 1: Already voted for candidate 1b75487eefe54cbdb4140dea8ab8a499 in this term.
I20251213 20:59:13.551697 12530 leader_election.cc:304] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5976941698547af85c98dff50a9d45b; no voters: 1b75487eefe54cbdb4140dea8ab8a499, 97f3d76e88294646bccacef46bb93c38
I20251213 20:59:13.552040 12748 raft_consensus.cc:3060] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:13.554754 12748 raft_consensus.cc:2749] T cd58b344007c431ab5d8771d201733b7 P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251213 20:59:13.566851 12733 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.60:36039
I20251213 20:59:13.566851 12850 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.60:36039 every 8 connection(s)
I20251213 20:59:13.568774 12733 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
I20251213 20:59:13.578454 12852 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:13.579268 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 12733
I20251213 20:59:13.583312 12852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap starting.
I20251213 20:59:13.593014 12852 log.cc:826] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:13.607246 12852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251213 20:59:13.608034 12852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap complete.
I20251213 20:59:13.616302 12852 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:13.617669 12852 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Initialized, Role: FOLLOWER
I20251213 20:59:13.618451 12852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:13.619297 12855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:13.619568 12855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:13.620733 12852 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:13.653005 12869 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:13.657307 12869 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: loading cluster ID for follower catalog manager: success
I20251213 20:59:13.662707 12869 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: acquiring CA information for follower catalog manager: success
I20251213 20:59:13.666240 12869 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:13.671502 12750 consensus_queue.cc:1048] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.673471 12733 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:13.679181 12733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:13.681218 12874 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 68c0fc24cab047dcaa4cae2dbd628626)
I20251213 20:59:13.681797 12874 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:13.683601 12501 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:13.683801 12501 heartbeater.cc:499] Master 127.11.111.60:36039 was elected leader, sending a full tablet report...
I20251213 20:59:13.687808 12816 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:34919; Asking this server to re-register.
I20251213 20:59:13.688830 12501 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:13.688851 12874 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:13.691530 12815 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
W20251213 20:59:13.712177 11969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20251213 20:59:13.720719 12884 consensus_queue.cc:1048] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.731233 12826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: false dest_uuid: "6736d1c95f384b9098c089a20b76e9ae" is_pre_election: true
I20251213 20:59:13.731941 12826 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fb6eff9fdc4139b18baa7e29377276 in term 3.
I20251213 20:59:13.732692 11968 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:13.733021 12874 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251213 20:59:13.733151 12874 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 68c0fc24cab047dcaa4cae2dbd628626)
I20251213 20:59:13.733247 12874 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 4 FOLLOWER]: Advancing to term 5
I20251213 20:59:13.736197 12874 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:13.737893 12874 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 5 election: Requested vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:13.738195 12826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: false dest_uuid: "6736d1c95f384b9098c089a20b76e9ae"
I20251213 20:59:13.738409 12826 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 3 FOLLOWER]: Advancing to term 5
W20251213 20:59:13.740756 11969 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:13.745347 12886 consensus_queue.cc:1048] T cb1f2d25f95c49b4918383371cc12368 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.745903 12886 consensus_queue.cc:1048] T 529995cf7f2e4b15b5e026178d59a916 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.750066 12886 consensus_queue.cc:1048] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.758491 12361 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:13.758731 12361 heartbeater.cc:499] Master 127.11.111.60:36039 was elected leader, sending a full tablet report...
I20251213 20:59:13.759672 12641 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:13.759877 12641 heartbeater.cc:499] Master 127.11.111.60:36039 was elected leader, sending a full tablet report...
I20251213 20:59:13.761219 12826 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fb6eff9fdc4139b18baa7e29377276 in term 5.
I20251213 20:59:13.761770 12815 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:48493; Asking this server to re-register.
I20251213 20:59:13.762593 12361 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:13.763597 11968 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:13.763983 12874 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 FOLLOWER]: Leader election won for term 5
I20251213 20:59:13.764161 12816 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
W20251213 20:59:13.765889 11969 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:13.771250 12748 consensus_queue.cc:1048] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.774123 12874 raft_consensus.cc:697] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 LEADER]: Becoming Leader. State: Replica: f2fb6eff9fdc4139b18baa7e29377276, State: Running, Role: LEADER
I20251213 20:59:13.774904 12874 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:13.778337 12894 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2fb6eff9fdc4139b18baa7e29377276. Latest consensus state: current_term: 5 leader_uuid: "f2fb6eff9fdc4139b18baa7e29377276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:13.779150 12894 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:13.786777 12750 consensus_queue.cc:1048] T 7d2e8c74b4ca427ab675b574516c1db1 P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.818953 12748 consensus_queue.cc:1048] T ff555a43366144fdb7f5c857d351ef3b P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.821806 12896 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:13.827533 12896 catalog_manager.cc:679] Loaded metadata for table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]
I20251213 20:59:13.828192 12896 catalog_manager.cc:679] Loaded metadata for table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]
I20251213 20:59:13.828774 12896 catalog_manager.cc:679] Loaded metadata for table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]
I20251213 20:59:13.829727 12896 catalog_manager.cc:679] Loaded metadata for table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]
I20251213 20:59:13.830199 12896 catalog_manager.cc:679] Loaded metadata for table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]
I20251213 20:59:13.830550 12816 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:50197; Asking this server to re-register.
I20251213 20:59:13.830699 12896 catalog_manager.cc:679] Loaded metadata for table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]
I20251213 20:59:13.831197 12896 catalog_manager.cc:679] Loaded metadata for table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]
I20251213 20:59:13.831673 12896 catalog_manager.cc:679] Loaded metadata for table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]
I20251213 20:59:13.831928 12641 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:13.832124 12896 catalog_manager.cc:679] Loaded metadata for table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]
I20251213 20:59:13.834796 12816 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:13.837198 12896 tablet_loader.cc:96] loaded metadata for tablet 529995cf7f2e4b15b5e026178d59a916 (table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261])
I20251213 20:59:13.837652 12896 tablet_loader.cc:96] loaded metadata for tablet 7d2e8c74b4ca427ab675b574516c1db1 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765])
I20251213 20:59:13.838009 12896 tablet_loader.cc:96] loaded metadata for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5])
I20251213 20:59:13.838594 12896 tablet_loader.cc:96] loaded metadata for tablet cb1f2d25f95c49b4918383371cc12368 (table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3])
I20251213 20:59:13.838977 12896 tablet_loader.cc:96] loaded metadata for tablet cd58b344007c431ab5d8771d201733b7 (table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3])
I20251213 20:59:13.839326 12896 tablet_loader.cc:96] loaded metadata for tablet da4c8619ce49439cb7b84eb17d587fb4 (table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60])
I20251213 20:59:13.839690 12896 tablet_loader.cc:96] loaded metadata for tablet f61276e680a24502bd0d0547a55b2abd (table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72])
I20251213 20:59:13.840034 12896 tablet_loader.cc:96] loaded metadata for tablet fa028486f240481f9081ea514153e384 (table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18])
I20251213 20:59:13.840380 12896 tablet_loader.cc:96] loaded metadata for tablet ff555a43366144fdb7f5c857d351ef3b (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153])
I20251213 20:59:13.840658 12896 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:13.841760 12896 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:13.841835 12896 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:13.843034 12896 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:13.843865 12896 catalog_manager.cc:6038] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Loaded TSK: 0
I20251213 20:59:13.844739 12896 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:13.845636 12896 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:13.846424 12896 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:13.872004 11980 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52802:
name: "default.Table_cc9c8f278dee46108580b4455e30b213"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:13.872341 11979 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_12be436a89df4cfcac0b036dbab2d770" } modify_external_catalogs: true
I20251213 20:59:13.872452 11978 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52802:
name: "default.Table_32850fadebae44e797c016b5161038ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:13.872714 11979 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_12be436a89df4cfcac0b036dbab2d770" } modify_external_catalogs: true
I20251213 20:59:13.873085 11977 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb" } modify_external_catalogs: true
I20251213 20:59:13.873198 11982 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52802:
name: "default.Table_29cded03ad224e8d809f516d66e903dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:13.873172 11976 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_4ece212e4b3b44968882ae1e8ca68422" } modify_external_catalogs: true
I20251213 20:59:13.873522 11976 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_4ece212e4b3b44968882ae1e8ca68422" } modify_external_catalogs: true
I20251213 20:59:13.873328 11977 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb" } modify_external_catalogs: true
I20251213 20:59:13.874265 11981 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52802:
name: "default.Table_9857f31a4d1e42b78febc6e2e68587fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:13.875131 11975 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52802:
name: "default.Table_60f4a42abdd34e9fbfa1e25215d9fba5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:13.875809 11974 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_e09d8c7259974a359a976428a2b351bb" } modify_external_catalogs: true
I20251213 20:59:13.876010 11974 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52802:
table { table_name: "default.Table_e09d8c7259974a359a976428a2b351bb" } modify_external_catalogs: true
I20251213 20:59:13.876773 11973 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d2e8c74b4ca427ab675b574516c1db1 from {username='slave'} at 127.0.0.1:52802
F20251213 20:59:13.881798 12194 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_ebe9a0444b8a40c7ac9d635be30c28b0, message={"server":"","servicePrincipal":"","db":"default","table":"table_ebe9a0444b8a40c7ac9d635be30c28b0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ebe9a0444b8a40c7ac9d635be30c28b0\",\"kudu.table_id\":\"2d6fd96071954bcb92508900e2a386b6\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_b027752248784b2f9fff90508e2b62b4, message={"server":"","servicePrincipal":"","db":"default","table":"table_b027752248784b2f9fff90508e2b62b4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b027752248784b2f9fff90508e2b62b4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_b027752248784b2f9fff90508e2b62b4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b027752248784b2f9fff90508e2b62b4\",\"kudu.table_id\":\"abe1a66f87914e24b17972fa42e24765\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7ff1c9542dcd google::LogMessage::Fail() at ??:0
I20251213 20:59:13.886047 12826 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Refusing update from remote peer f2fb6eff9fdc4139b18baa7e29377276: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
@ 0x7ff1c9546b93 google::LogMessage::SendToLog() at ??:0
W20251213 20:59:13.886646 11969 consensus_peers.cc:597] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 -> Peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Couldn't send request to peer 68c0fc24cab047dcaa4cae2dbd628626. Status: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251213 20:59:13.886945 12894 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 5, Time since last communication: 0.000s
I20251213 20:59:13.887121 12894 consensus_queue.cc:1243] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [LEADER]: Peer 6736d1c95f384b9098c089a20b76e9ae log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
@ 0x7ff1c95427cc google::LogMessage::Flush() at ??:0
@ 0x7ff1c9542a11 google::LogMessage::~LogMessage() at ??:0
I20251213 20:59:13.889015 12826 pending_rounds.cc:85] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Aborting all ops after (but not including) 5
I20251213 20:59:13.889127 12826 pending_rounds.cc:107] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251213 20:59:13.889226 12826 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
@ 0x7ff1dff28aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7ff1dff27351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ff1dff2efcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7ff1dff2ee49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7ff1cc2b73f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff1cf8c26db start_thread at ??:0
@ 0x7ff1c83f371f clone at ??:0
I20251213 20:59:13.931000 12909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "f2fb6eff9fdc4139b18baa7e29377276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:13.931378 12909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:13.931691 12855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2fb6eff9fdc4139b18baa7e29377276. Latest consensus state: current_term: 5 leader_uuid: "f2fb6eff9fdc4139b18baa7e29377276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:13.931891 12855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:13.942576 12905 mvcc.cc:204] Tried to move back new op lower bound from 7232141529633112064 to 7232141524746899456. Current Snapshot: MvccSnapshot[applied={T|T < 7232141529633112064}]
I20251213 20:59:13.990253 12719 consensus_queue.cc:1048] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:13.999262 12719 consensus_queue.cc:1048] T cd58b344007c431ab5d8771d201733b7 P 1b75487eefe54cbdb4140dea8ab8a499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.030023 12748 consensus_queue.cc:1048] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.037247 12748 consensus_queue.cc:1048] T da4c8619ce49439cb7b84eb17d587fb4 P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.050019 12748 consensus_queue.cc:1048] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.057963 12901 consensus_queue.cc:1048] T 7f43dfbead524dc6aa1cddfbafe6e639 P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.062929 12750 raft_consensus.cc:493] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:14.063105 12750 raft_consensus.cc:515] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:14.063750 12750 leader_election.cc:290] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:14.064394 12594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f61276e680a24502bd0d0547a55b2abd" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b" is_pre_election: true
I20251213 20:59:14.064615 12594 raft_consensus.cc:2468] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:14.065034 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f61276e680a24502bd0d0547a55b2abd" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499" is_pre_election: true
I20251213 20:59:14.065258 12456 raft_consensus.cc:2468] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 0.
I20251213 20:59:14.065806 12251 leader_election.cc:304] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:14.066077 12750 raft_consensus.cc:2804] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251213 20:59:14.066205 12750 raft_consensus.cc:493] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:14.066291 12750 raft_consensus.cc:3060] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:14.069144 12750 raft_consensus.cc:515] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:14.070415 12456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f61276e680a24502bd0d0547a55b2abd" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b75487eefe54cbdb4140dea8ab8a499"
I20251213 20:59:14.070659 12594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f61276e680a24502bd0d0547a55b2abd" candidate_uuid: "97f3d76e88294646bccacef46bb93c38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5976941698547af85c98dff50a9d45b"
I20251213 20:59:14.070745 12456 raft_consensus.cc:3060] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499 [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:14.070845 12594 raft_consensus.cc:3060] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b [term 0 FOLLOWER]: Advancing to term 1
I20251213 20:59:14.073827 12456 raft_consensus.cc:2468] T f61276e680a24502bd0d0547a55b2abd P 1b75487eefe54cbdb4140dea8ab8a499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:14.073829 12594 raft_consensus.cc:2468] T f61276e680a24502bd0d0547a55b2abd P c5976941698547af85c98dff50a9d45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f3d76e88294646bccacef46bb93c38 in term 1.
I20251213 20:59:14.074383 12251 leader_election.cc:304] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f3d76e88294646bccacef46bb93c38, c5976941698547af85c98dff50a9d45b; no voters:
I20251213 20:59:14.074771 12886 raft_consensus.cc:2804] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 1 FOLLOWER]: Leader election won for term 1
I20251213 20:59:14.074952 12886 raft_consensus.cc:697] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [term 1 LEADER]: Becoming Leader. State: Replica: 97f3d76e88294646bccacef46bb93c38, State: Running, Role: LEADER
I20251213 20:59:14.075248 12886 consensus_queue.cc:237] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 } } peers { permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 } } peers { permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 } }
I20251213 20:59:14.078250 12750 leader_election.cc:290] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [CANDIDATE]: Term 1 election: Requested vote from peers 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023), c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
W20251213 20:59:14.131100 12530 connection.cc:537] client connection to 127.11.111.62:33723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:14.131649 12390 connection.cc:537] client connection to 127.11.111.62:33723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:14.133277 12250 connection.cc:537] client connection to 127.11.111.62:33723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:14.134830 12724 negotiation.cc:337] Failed RPC negotiation. Trace:
1213 20:59:14.132495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.111.62:33723 (local address 127.11.111.2:32841)
1213 20:59:14.132727 (+ 232us) negotiation.cc:107] Waiting for socket to connect
1213 20:59:14.132745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1213 20:59:14.132859 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1213 20:59:14.133841 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129}
W20251213 20:59:14.136528 12801 connection.cc:537] server connection from 127.0.0.1:39222 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:14.137881 12892 negotiation.cc:337] Failed RPC negotiation. Trace:
1213 20:59:14.132021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.111.62:33723 (local address 127.11.111.3:49809)
1213 20:59:14.132317 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1213 20:59:14.132341 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1213 20:59:14.132509 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1213 20:59:14.137002 (+ 4493us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":171}
I20251213 20:59:14.307700 12901 consensus_queue.cc:1048] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f3d76e88294646bccacef46bb93c38" member_type: VOTER last_known_addr { host: "127.11.111.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.311515 12748 consensus_queue.cc:1048] T fa028486f240481f9081ea514153e384 P c5976941698547af85c98dff50a9d45b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.390293 12855 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f2fb6eff9fdc4139b18baa7e29377276)
I20251213 20:59:14.390493 12855 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:14.392175 12855 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
W20251213 20:59:14.393168 12801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:14.394268 12803 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:14.394456 12801 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:14.394582 12801 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae; no voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:14.394910 12855 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251213 20:59:14.899288 12942 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f2fb6eff9fdc4139b18baa7e29377276)
I20251213 20:59:14.899473 12942 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:14.900142 12942 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
W20251213 20:59:14.902164 12801 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:14.902343 12803 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:14.902597 12803 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae; no voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:14.902827 12942 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251213 20:59:14.974225 12940 consensus_queue.cc:1048] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5976941698547af85c98dff50a9d45b" member_type: VOTER last_known_addr { host: "127.11.111.3" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:14.979176 12941 consensus_queue.cc:1048] T f61276e680a24502bd0d0547a55b2abd P 97f3d76e88294646bccacef46bb93c38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" member_type: VOTER last_known_addr { host: "127.11.111.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251213 20:59:15.608814 12959 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f2fb6eff9fdc4139b18baa7e29377276)
I20251213 20:59:15.608985 12959 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:15.609656 12959 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
W20251213 20:59:15.611290 12801 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:15.611505 12803 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:15.611636 12803 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae; no voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:15.611869 12959 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251213 20:59:16.308097 12974 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f2fb6eff9fdc4139b18baa7e29377276)
I20251213 20:59:16.308336 12974 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:16.308972 12974 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
W20251213 20:59:16.310802 12803 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:16.310964 12801 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:16.311084 12801 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae; no voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:16.311491 12974 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251213 20:59:16.743336 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--webserver_interface=127.11.111.61
--webserver_port=42211
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251213 20:59:17.081894 12985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:17.082208 12985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:17.082293 12985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:17.090641 12985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:17.090745 12985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:17.090806 12985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:17.090864 12985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:17.101049 12985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.111.61
--webserver_port=42211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:17.102682 12985 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:17.104493 12985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:17.112884 12996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:17.113231 12995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:17.113952 12998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:17.115482 12985 server_base.cc:1047] running on GCE node
I20251213 20:59:17.116302 12985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:17.117986 12985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:17.119210 12985 hybrid_clock.cc:648] HybridClock initialized: now 1765659557119147 us; error 56 us; skew 500 ppm
I20251213 20:59:17.119596 12985 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:17.121858 12985 webserver.cc:492] Webserver started at http://127.11.111.61:42211/ using document root <none> and password file <none>
I20251213 20:59:17.122606 12985 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:17.122722 12985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:17.128378 12985 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251213 20:59:17.131520 13004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:17.132792 12985 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251213 20:59:17.132946 12985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
uuid: "68c0fc24cab047dcaa4cae2dbd628626"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:17.133687 12985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:17.151310 12985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:17.152113 12985 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:17.152374 12985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:17.170717 12985 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.61:39203
I20251213 20:59:17.170753 13055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.61:39203 every 8 connection(s)
I20251213 20:59:17.172309 12985 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
I20251213 20:59:17.177675 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 12985
I20251213 20:59:17.180006 13056 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:17.183588 13056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap starting.
I20251213 20:59:17.194597 13056 log.cc:826] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:17.204668 12503 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:17.223382 13056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=23 ignored=0} mutations{seen=18 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251213 20:59:17.223909 13056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap complete.
I20251213 20:59:17.229146 13056 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:17.229930 13056 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Initialized, Role: FOLLOWER
I20251213 20:59:17.230510 13056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:17.231225 13062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:17.231470 13062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:17.232213 13056 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:17.241333 12985 master_runner.cc:186] Error getting master registration for 127.11.111.62:33723: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111),
I20251213 20:59:17.245126 12985 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:17.249475 13076 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:17.249589 13076 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: loading cluster ID for follower catalog manager: success
I20251213 20:59:17.253860 13076 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: acquiring CA information for follower catalog manager: success
I20251213 20:59:17.256764 13076 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:17.258061 13080 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f2fb6eff9fdc4139b18baa7e29377276)
I20251213 20:59:17.258255 13080 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:17.259021 13080 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
W20251213 20:59:17.262197 12801 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:17.266691 13031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:17.267234 13031 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 4.
I20251213 20:59:17.267795 12803 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626; no voters: f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:17.268102 13080 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251213 20:59:17.268229 13080 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Starting leader election (detected failure of leader f2fb6eff9fdc4139b18baa7e29377276)
I20251213 20:59:17.268324 13080 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 5 FOLLOWER]: Advancing to term 6
I20251213 20:59:17.271241 13080 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:17.272022 13080 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:17.272608 13031 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626"
I20251213 20:59:17.272842 13031 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 4 FOLLOWER]: Advancing to term 6
W20251213 20:59:17.273424 12801 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:17.277029 13031 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 6.
I20251213 20:59:17.277544 12803 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626; no voters: f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:17.277866 13080 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Leader election won for term 6
I20251213 20:59:17.278363 13080 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 LEADER]: Becoming Leader. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Running, Role: LEADER
I20251213 20:59:17.278913 13080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.32, Last appended by leader: 32, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:17.281033 13084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 6 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:17.281319 13084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:17.389446 12643 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:17.391942 13021 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:43143; Asking this server to re-register.
I20251213 20:59:17.392663 12643 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:17.394493 13021 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:17.594183 12363 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:17.596777 13021 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:46797; Asking this server to re-register.
I20251213 20:59:17.597710 12363 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:17.599495 13021 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:17.710318 13021 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:38163; Asking this server to re-register.
I20251213 20:59:17.711118 12503 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:17.712661 13021 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
W20251213 20:59:17.745863 12801 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae -> Peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Couldn't send request to peer f2fb6eff9fdc4139b18baa7e29377276. Status: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251213 20:59:17.905411 13031 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Refusing update from remote peer 6736d1c95f384b9098c089a20b76e9ae: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20251213 20:59:17.906378 13084 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 30, Time since last communication: 0.000s
I20251213 20:59:17.912048 13090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 6 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:17.912339 13090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:17.915441 13090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:17.915386 13098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:17.915688 13090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:17.915696 13098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:17.916422 13084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:17.916687 13084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:17.916754 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:17.916999 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:17.926648 13086 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:17.929443 13086 catalog_manager.cc:679] Loaded metadata for table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]
I20251213 20:59:17.930124 13086 catalog_manager.cc:679] Loaded metadata for table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]
I20251213 20:59:17.930604 13086 catalog_manager.cc:679] Loaded metadata for table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]
I20251213 20:59:17.930976 13086 catalog_manager.cc:679] Loaded metadata for table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]
I20251213 20:59:17.931363 13086 catalog_manager.cc:679] Loaded metadata for table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]
I20251213 20:59:17.931740 13086 catalog_manager.cc:679] Loaded metadata for table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]
I20251213 20:59:17.932027 13086 catalog_manager.cc:679] Loaded metadata for table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]
I20251213 20:59:17.932322 13086 catalog_manager.cc:679] Loaded metadata for table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]
I20251213 20:59:17.932600 13086 catalog_manager.cc:679] Loaded metadata for table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]
I20251213 20:59:17.934753 13086 tablet_loader.cc:96] loaded metadata for tablet 529995cf7f2e4b15b5e026178d59a916 (table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261])
I20251213 20:59:17.935232 13086 tablet_loader.cc:96] loaded metadata for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5])
I20251213 20:59:17.935716 13086 tablet_loader.cc:96] loaded metadata for tablet cb1f2d25f95c49b4918383371cc12368 (table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3])
I20251213 20:59:17.935959 13086 tablet_loader.cc:96] loaded metadata for tablet cd58b344007c431ab5d8771d201733b7 (table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3])
I20251213 20:59:17.936213 13086 tablet_loader.cc:96] loaded metadata for tablet d03a7cadb5a044a99f6f18ca2ee4fb36 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765])
I20251213 20:59:17.936473 13086 tablet_loader.cc:96] loaded metadata for tablet da4c8619ce49439cb7b84eb17d587fb4 (table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60])
I20251213 20:59:17.936719 13086 tablet_loader.cc:96] loaded metadata for tablet f61276e680a24502bd0d0547a55b2abd (table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72])
I20251213 20:59:17.936944 13086 tablet_loader.cc:96] loaded metadata for tablet fa028486f240481f9081ea514153e384 (table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18])
I20251213 20:59:17.937180 13086 tablet_loader.cc:96] loaded metadata for tablet ff555a43366144fdb7f5c857d351ef3b (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153])
I20251213 20:59:17.937454 13086 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:17.938450 13086 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:17.938520 13086 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:17.939800 13086 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:17.940624 13086 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Loaded TSK: 0
I20251213 20:59:17.941525 13086 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:17.942363 13086 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:17.943135 13086 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:17.962726 12816 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39202:
table { table_name: "default.Table_1368482eba894f4c8b68346432f132d8" } new_table_name: "default.Table_3c93b9a42ca143889e8ab38b6fc45d47" modify_external_catalogs: true
I20251213 20:59:17.962934 12814 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39202:
table { table_name: "default.Table_b027752248784b2f9fff90508e2b62b4" } new_table_name: "default.Table_31c612ce45a347a2b23e54b30cf823be" modify_external_catalogs: true
I20251213 20:59:17.963115 12813 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39202:
name: "default.Table_5009d0ba1a394917a7101259634328ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:17.963532 12807 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39202:
name: "default.Table_32850fadebae44e797c016b5161038ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:17.963595 12812 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39202:
name: "default.Table_9e21ceee05264206866e8b0d62a537c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:17.964162 12811 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39202:
name: "default.Table_e448c113874f43c6b7a38f4b918a08a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:17.965011 12808 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39202:
table { table_name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb" } modify_external_catalogs: true
I20251213 20:59:17.965226 12808 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39202:
table { table_name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb" } modify_external_catalogs: true
I20251213 20:59:17.965617 12815 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39202:
table { table_name: "default.Table_0dc454009abd48db8c91e0aa3c76eb4c" } new_table_name: "default.Table_7c733924e7c54175b89a1b37e0095c4b" modify_external_catalogs: true
I20251213 20:59:17.965520 12810 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39202:
name: "default.Table_0667e63914504edaa41a24b9c799bfba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:17.965759 12809 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39202:
table { table_name: "default.Table_e15c8fc8dfd143aaa4b8d039e22cf63d" } new_table_name: "default.Table_40016a174bfc4e3c9f29ec786589b626" modify_external_catalogs: true
F20251213 20:59:17.969587 12868 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_ebe9a0444b8a40c7ac9d635be30c28b0, message={"server":"","servicePrincipal":"","db":"default","table":"table_ebe9a0444b8a40c7ac9d635be30c28b0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ebe9a0444b8a40c7ac9d635be30c28b0\",\"kudu.table_id\":\"2d6fd96071954bcb92508900e2a386b6\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_b027752248784b2f9fff90508e2b62b4, message={"server":"","servicePrincipal":"","db":"default","table":"table_b027752248784b2f9fff90508e2b62b4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b027752248784b2f9fff90508e2b62b4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_b027752248784b2f9fff90508e2b62b4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b027752248784b2f9fff90508e2b62b4\",\"kudu.table_id\":\"abe1a66f87914e24b17972fa42e24765\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f04e6cc1dcd google::LogMessage::Fail() at ??:0
@ 0x7f04e6cc5b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f04e6cc17cc google::LogMessage::Flush() at ??:0
@ 0x7f04e6cc1a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f04fd6a7aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f04fd6a6351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f04fd6adfcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f04fd6ade49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f04e9a363f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f04ed0416db start_thread at ??:0
@ 0x7f04e5b7271f clone at ??:0
W20251213 20:59:18.164209 12251 connection.cc:537] client connection to 127.11.111.60:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:18.165154 12650 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:18.165691 12650 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:18.165925 12650 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.111.60:36039)
W20251213 20:59:18.166435 12650 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.111.60:36039)
I20251213 20:59:18.414321 13098 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:18.414542 13098 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:18.416504 13098 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:18.417425 13006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:18.418581 13006 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:18.418790 13007 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:18.418926 13007 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:18.419257 13098 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251213 20:59:19.015870 13126 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:19.016053 13126 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:19.016705 13126 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:19.018767 13007 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:19.018988 13006 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:19.019107 13006 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:19.019363 13126 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251213 20:59:19.497749 11708 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskli_hB8/build/asan/bin/kudu pid 12985
I20251213 20:59:19.507921 13051 generic_service.cc:196] Checking for leaks (request via RPC)
I20251213 20:59:19.618322 11708 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskli_hB8/build/asan/bin/kudu with pid 12985
I20251213 20:59:20.409744 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--webserver_interface=127.11.111.61
--webserver_port=42211
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251213 20:59:20.763693 13146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:20.764068 13146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:20.764179 13146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:20.772718 13146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:20.772850 13146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:20.772931 13146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:20.772996 13146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:20.783245 13146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.111.61
--webserver_port=42211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:20.785022 13146 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:20.787292 13146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:20.795850 13152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:20.796015 13155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:20.795821 13153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:20.796877 13146 server_base.cc:1047] running on GCE node
I20251213 20:59:20.797744 13146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:20.799201 13146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:20.800410 13146 hybrid_clock.cc:648] HybridClock initialized: now 1765659560800359 us; error 46 us; skew 500 ppm
I20251213 20:59:20.800779 13146 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:20.803119 13146 webserver.cc:492] Webserver started at http://127.11.111.61:42211/ using document root <none> and password file <none>
I20251213 20:59:20.804041 13146 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:20.804163 13146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:20.809899 13146 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251213 20:59:20.813014 13161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:20.814507 13146 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251213 20:59:20.814677 13146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
uuid: "68c0fc24cab047dcaa4cae2dbd628626"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:20.815368 13146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:20.850030 13146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:20.850987 13146 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:20.851269 13146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:20.870540 13146 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.61:39203
I20251213 20:59:20.870565 13212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.61:39203 every 8 connection(s)
I20251213 20:59:20.872227 13146 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
I20251213 20:59:20.877879 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 13146
I20251213 20:59:20.879738 13213 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:20.883427 13213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap starting.
I20251213 20:59:20.895807 13213 log.cc:826] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:20.914367 12643 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:20.925323 13213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251213 20:59:20.926014 13213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap complete.
I20251213 20:59:20.933466 13213 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:20.934486 13213 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Initialized, Role: FOLLOWER
I20251213 20:59:20.935252 13213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:20.936286 13218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:20.936558 13218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:20.937510 13213 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:20.951232 13232 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:20.951685 13232 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: loading cluster ID for follower catalog manager: success
I20251213 20:59:20.947291 13146 master_runner.cc:186] Error getting master registration for 127.11.111.62:33723: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111),
I20251213 20:59:20.957796 13146 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:20.961083 13232 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: acquiring CA information for follower catalog manager: success
I20251213 20:59:20.964288 13232 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:21.146373 12363 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:21.148898 13178 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:39711; Asking this server to re-register.
I20251213 20:59:21.149696 12363 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:21.151820 13178 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:21.234007 12503 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:21.235823 13177 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:55189; Asking this server to re-register.
I20251213 20:59:21.236541 12503 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:21.238045 13178 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:21.418509 13178 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:58495; Asking this server to re-register.
I20251213 20:59:21.419173 12643 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:21.420713 13178 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:21.559888 13238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:21.560150 13238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:21.562106 13238 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:21.563110 13164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:21.564324 13163 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:21.564486 13164 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:21.564649 13164 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:21.564996 13238 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251213 20:59:22.223237 13246 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:22.223436 13246 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:22.224174 13246 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:22.226133 13164 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:22.226344 13163 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:22.226455 13163 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:22.226701 13246 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251213 20:59:22.834082 13255 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:22.834331 13255 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:22.835029 13255 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:22.837321 13164 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:22.837508 13163 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:22.837671 13163 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:22.837952 13255 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251213 20:59:23.752183 13269 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:23.752388 13269 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:23.752950 13269 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:23.755039 13163 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:23.755213 13164 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:23.755347 13164 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:23.755574 13269 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251213 20:59:23.957787 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--webserver_interface=127.11.111.60
--webserver_port=36499
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251213 20:59:24.294728 13275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:24.295100 13275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:24.295213 13275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:24.303712 13275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:24.303835 13275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:24.303907 13275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:24.303968 13275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:24.314663 13275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.111.60
--webserver_port=36499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:24.316324 13275 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:24.318197 13275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:24.326510 13282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:24.326581 13283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:24.327029 13285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:24.327569 13275 server_base.cc:1047] running on GCE node
I20251213 20:59:24.328351 13275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:24.329878 13275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:24.331091 13275 hybrid_clock.cc:648] HybridClock initialized: now 1765659564331034 us; error 57 us; skew 500 ppm
I20251213 20:59:24.331440 13275 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:24.333774 13275 webserver.cc:492] Webserver started at http://127.11.111.60:36499/ using document root <none> and password file <none>
I20251213 20:59:24.334601 13275 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:24.334705 13275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:24.340358 13275 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251213 20:59:24.343636 13291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:24.345057 13275 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251213 20:59:24.345211 13275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
uuid: "6736d1c95f384b9098c089a20b76e9ae"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:24.345975 13275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:24.366562 13275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:24.367502 13275 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:24.367780 13275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:24.387503 13275 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.60:36039
I20251213 20:59:24.387526 13342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.60:36039 every 8 connection(s)
I20251213 20:59:24.389140 13275 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
I20251213 20:59:24.395654 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 13275
I20251213 20:59:24.396456 13343 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:24.400382 13343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap starting.
I20251213 20:59:24.414170 13343 log.cc:826] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:24.417675 12641 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:24.440066 13343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=1 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251213 20:59:24.440649 13343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap complete.
I20251213 20:59:24.445869 13343 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:24.446543 13343 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Initialized, Role: FOLLOWER
I20251213 20:59:24.447175 13343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:24.447934 13350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:24.448251 13350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:24.448905 13343 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:24.462976 13275 master_runner.cc:186] Error getting master registration for 127.11.111.62:33723: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111),
I20251213 20:59:24.469599 13275 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:24.473126 13364 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:24.473219 13364 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: loading cluster ID for follower catalog manager: success
I20251213 20:59:24.476198 13364 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: acquiring CA information for follower catalog manager: success
I20251213 20:59:24.478374 13364 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:24.686343 12361 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:24.688540 13307 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:51735; Asking this server to re-register.
I20251213 20:59:24.689241 12361 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:24.691123 13307 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:24.730041 12501 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:24.731459 13307 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:40267; Asking this server to re-register.
I20251213 20:59:24.732007 12501 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:24.733510 13307 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:24.921855 13307 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:46783; Asking this server to re-register.
I20251213 20:59:24.922516 12641 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:24.924113 13307 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:24.964329 13371 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:24.964624 13371 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:24.966663 13371 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
W20251213 20:59:24.967708 13293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:24.972710 13293 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:24.973194 13188 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 7 candidate_status { last_received { term: 6 index: 33 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:24.973644 13188 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 6.
I20251213 20:59:24.974275 13295 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626; no voters: f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:24.974669 13371 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251213 20:59:24.974815 13371 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:24.974910 13371 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 6 FOLLOWER]: Advancing to term 7
I20251213 20:59:24.978475 13371 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:24.979100 13371 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 7 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:24.979635 13188 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 7 candidate_status { last_received { term: 6 index: 33 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626"
I20251213 20:59:24.979848 13188 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 6 FOLLOWER]: Advancing to term 7
W20251213 20:59:24.980283 13293 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:24.983479 13188 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 7.
I20251213 20:59:24.983865 13295 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626; no voters: f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:24.984159 13371 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 FOLLOWER]: Leader election won for term 7
I20251213 20:59:24.984576 13371 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 LEADER]: Becoming Leader. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Running, Role: LEADER
I20251213 20:59:24.985005 13371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:24.987687 13376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 7 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:24.987938 13376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:24.988520 13378 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:24.991011 13378 catalog_manager.cc:679] Loaded metadata for table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]
I20251213 20:59:24.991421 13378 catalog_manager.cc:679] Loaded metadata for table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]
I20251213 20:59:24.991756 13378 catalog_manager.cc:679] Loaded metadata for table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]
I20251213 20:59:24.992067 13378 catalog_manager.cc:679] Loaded metadata for table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]
I20251213 20:59:24.992360 13378 catalog_manager.cc:679] Loaded metadata for table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]
I20251213 20:59:24.992676 13378 catalog_manager.cc:679] Loaded metadata for table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]
I20251213 20:59:24.992970 13378 catalog_manager.cc:679] Loaded metadata for table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]
I20251213 20:59:24.993292 13378 catalog_manager.cc:679] Loaded metadata for table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]
I20251213 20:59:24.993667 13378 catalog_manager.cc:679] Loaded metadata for table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]
I20251213 20:59:24.995604 13378 tablet_loader.cc:96] loaded metadata for tablet 529995cf7f2e4b15b5e026178d59a916 (table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261])
I20251213 20:59:24.996065 13378 tablet_loader.cc:96] loaded metadata for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5])
I20251213 20:59:24.996495 13378 tablet_loader.cc:96] loaded metadata for tablet cb1f2d25f95c49b4918383371cc12368 (table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3])
I20251213 20:59:24.996778 13378 tablet_loader.cc:96] loaded metadata for tablet cd58b344007c431ab5d8771d201733b7 (table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3])
I20251213 20:59:24.997010 13378 tablet_loader.cc:96] loaded metadata for tablet d03a7cadb5a044a99f6f18ca2ee4fb36 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765])
I20251213 20:59:24.997279 13378 tablet_loader.cc:96] loaded metadata for tablet da4c8619ce49439cb7b84eb17d587fb4 (table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60])
I20251213 20:59:24.997596 13378 tablet_loader.cc:96] loaded metadata for tablet f61276e680a24502bd0d0547a55b2abd (table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72])
I20251213 20:59:24.997862 13378 tablet_loader.cc:96] loaded metadata for tablet fa028486f240481f9081ea514153e384 (table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18])
I20251213 20:59:24.998148 13378 tablet_loader.cc:96] loaded metadata for tablet ff555a43366144fdb7f5c857d351ef3b (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153])
I20251213 20:59:24.998391 13378 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:24.999380 13378 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:24.999451 13378 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:25.000689 13378 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:25.001631 13378 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Loaded TSK: 0
I20251213 20:59:25.002516 13378 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:25.003481 13378 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:25.004276 13378 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:25.048224 13307 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "default.Table_b7f4c624bfff43c8a5177f45a23ff719" } new_table_name: "default.Table_cbfd66dd739f453f8b64a7f6eaf0b21b" modify_external_catalogs: true
I20251213 20:59:25.048363 13308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52974:
name: "default.Table_eb562e37a81541048c6a9757feca4265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:25.048672 13303 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d2e8c74b4ca427ab675b574516c1db1 from {username='slave'} at 127.0.0.1:52974
I20251213 20:59:25.049278 13306 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d2e8c74b4ca427ab675b574516c1db1 from {username='slave'} at 127.0.0.1:52974
I20251213 20:59:25.049796 13299 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52974:
name: "default.Table_9e21ceee05264206866e8b0d62a537c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:25.049997 13306 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "default.Table_b027752248784b2f9fff90508e2b62b4" } new_table_name: "default.Table_31c612ce45a347a2b23e54b30cf823be" modify_external_catalogs: true
I20251213 20:59:25.049965 13303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52974:
name: "default.Table_5009d0ba1a394917a7101259634328ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:25.050428 13305 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d2e8c74b4ca427ab675b574516c1db1 from {username='slave'} at 127.0.0.1:52974
I20251213 20:59:25.050479 13304 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d2e8c74b4ca427ab675b574516c1db1 from {username='slave'} at 127.0.0.1:52974
I20251213 20:59:25.050510 13302 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d2e8c74b4ca427ab675b574516c1db1 from {username='slave'} at 127.0.0.1:52974
I20251213 20:59:25.051052 13305 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "default.Table_0dc454009abd48db8c91e0aa3c76eb4c" } new_table_name: "default.Table_7c733924e7c54175b89a1b37e0095c4b" modify_external_catalogs: true
I20251213 20:59:25.051384 13304 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "default.Table_1368482eba894f4c8b68346432f132d8" } new_table_name: "default.Table_3c93b9a42ca143889e8ab38b6fc45d47" modify_external_catalogs: true
I20251213 20:59:25.051636 13300 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52974:
name: "default.Table_e448c113874f43c6b7a38f4b918a08a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:25.052006 13301 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "default.Table_e09d8c7259974a359a976428a2b351bb" } modify_external_catalogs: true
I20251213 20:59:25.052258 13301 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52974:
table { table_name: "default.Table_e09d8c7259974a359a976428a2b351bb" } modify_external_catalogs: true
I20251213 20:59:25.052599 13302 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52974:
name: "default.Table_0667e63914504edaa41a24b9c799bfba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251213 20:59:25.055971 13363 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_ebe9a0444b8a40c7ac9d635be30c28b0, message={"server":"","servicePrincipal":"","db":"default","table":"table_ebe9a0444b8a40c7ac9d635be30c28b0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ebe9a0444b8a40c7ac9d635be30c28b0\",\"kudu.table_id\":\"2d6fd96071954bcb92508900e2a386b6\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_b027752248784b2f9fff90508e2b62b4, message={"server":"","servicePrincipal":"","db":"default","table":"table_b027752248784b2f9fff90508e2b62b4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b027752248784b2f9fff90508e2b62b4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_b027752248784b2f9fff90508e2b62b4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b027752248784b2f9fff90508e2b62b4\",\"kudu.table_id\":\"abe1a66f87914e24b17972fa42e24765\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f7b961dfdcd google::LogMessage::Fail() at ??:0
@ 0x7f7b961e3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f7b961df7cc google::LogMessage::Flush() at ??:0
@ 0x7f7b961dfa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f7bacbc5aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f7bacbc4351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f7bacbcbfcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f7bacbcbe49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f7b98f543f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7b9c55f6db start_thread at ??:0
@ 0x7f7b9509071f clone at ??:0
W20251213 20:59:25.253983 12391 connection.cc:537] client connection to 127.11.111.60:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:25.254308 12251 connection.cc:537] client connection to 127.11.111.60:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:25.256178 12650 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:25.256491 12650 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:25.257211 12650 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:25.259171 12650 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.111.60:36039)
I20251213 20:59:25.959362 13146 master_runner.cc:186] Error getting master registration for 127.11.111.62:33723: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111),
I20251213 20:59:25.959579 13146 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:26.512446 13397 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:26.512651 13397 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:26.513340 13397 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:26.515358 13164 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:26.515517 13163 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:26.515653 13163 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:26.515914 13397 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:26.526381 11708 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskli_hB8/build/asan/bin/kudu pid 13146
I20251213 20:59:26.536008 13208 generic_service.cc:196] Checking for leaks (request via RPC)
I20251213 20:59:26.646953 11708 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskli_hB8/build/asan/bin/kudu with pid 13146
I20251213 20:59:27.518385 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--webserver_interface=127.11.111.61
--webserver_port=42211
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251213 20:59:27.854346 13406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:27.854660 13406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:27.854766 13406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:27.863112 13406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:27.863219 13406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:27.863298 13406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:27.863363 13406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:27.874081 13406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.111.61
--webserver_port=42211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:27.875842 13406 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:27.877750 13406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:27.886292 13415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:27.886188 13412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:27.886294 13413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:27.887140 13406 server_base.cc:1047] running on GCE node
I20251213 20:59:27.887939 13406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:27.889461 13406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:27.890694 13406 hybrid_clock.cc:648] HybridClock initialized: now 1765659567890625 us; error 74 us; skew 500 ppm
I20251213 20:59:27.891040 13406 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:27.893275 13406 webserver.cc:492] Webserver started at http://127.11.111.61:42211/ using document root <none> and password file <none>
I20251213 20:59:27.894112 13406 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:27.894207 13406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:27.899905 13406 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20251213 20:59:27.904589 13421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:27.906261 13406 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251213 20:59:27.906432 13406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
uuid: "68c0fc24cab047dcaa4cae2dbd628626"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:27.907181 13406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:27.933591 13406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:27.934556 13406 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:27.934800 13406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:27.953919 13406 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.61:39203
I20251213 20:59:27.953940 13472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.61:39203 every 8 connection(s)
I20251213 20:59:27.955564 13406 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
I20251213 20:59:27.955937 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 13406
I20251213 20:59:27.963143 13473 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:27.967000 13473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap starting.
I20251213 20:59:27.978274 13473 log.cc:826] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:28.004520 13473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251213 20:59:28.005110 13473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap complete.
I20251213 20:59:28.010422 13473 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:28.011162 13473 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Initialized, Role: FOLLOWER
I20251213 20:59:28.011809 13473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:28.012717 13477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:28.014443 13477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:28.014456 13473 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:28.024952 13406 master_runner.cc:186] Error getting master registration for 127.11.111.62:33723: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111),
I20251213 20:59:28.031832 13406 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:28.032931 13491 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:28.033010 13491 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: loading cluster ID for follower catalog manager: success
I20251213 20:59:28.036226 13491 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: acquiring CA information for follower catalog manager: success
I20251213 20:59:28.038365 13491 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:28.188408 12363 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:28.190973 13438 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:33727; Asking this server to re-register.
I20251213 20:59:28.191692 12363 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:28.193638 13438 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:28.270920 12503 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:28.272198 13438 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:56529; Asking this server to re-register.
I20251213 20:59:28.272794 12503 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:28.274308 13438 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:28.443967 13477 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:28.444226 13477 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:28.446215 13477 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:28.446977 13423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:28.448191 13424 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:28.448462 13423 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:28.448633 13423 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:28.449036 13477 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:28.453528 12643 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:28.454844 13438 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:39847; Asking this server to re-register.
I20251213 20:59:28.455312 12643 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:28.456477 13438 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:29.142390 13499 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:29.142621 13499 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:29.143384 13499 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:29.145730 13423 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:29.145885 13424 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:29.146109 13424 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:29.146488 13499 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:30.041252 13505 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:30.041498 13505 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:30.042130 13505 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:30.044126 13423 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:30.044279 13424 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:30.044418 13424 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:30.044662 13505 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:30.730610 13510 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:30.730814 13510 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:30.731491 13510 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:30.733568 13423 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
W20251213 20:59:30.733752 13424 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:30.733901 13424 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:30.734145 13510 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:31.307101 13515 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:31.307279 13515 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:31.307953 13515 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:31.310973 13424 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:31.311139 13423 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723): Network error: Client connection negotiation failed: client connection to 127.11.111.62:33723: connect: Connection refused (error 111)
I20251213 20:59:31.311260 13423 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626; no voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:31.311515 13515 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:31.333357 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.62:33723
--webserver_interface=127.11.111.62
--webserver_port=44033
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251213 20:59:31.658816 13519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:31.659139 13519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:31.659260 13519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:31.667776 13519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:31.667868 13519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:31.667918 13519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:31.667963 13519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:31.677604 13519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.62:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.111.62
--webserver_port=44033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:31.679154 13519 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:31.680847 13519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:31.688441 13526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:31.688417 13525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:31.689538 13528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:31.689625 13519 server_base.cc:1047] running on GCE node
I20251213 20:59:31.690358 13519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:31.691727 13519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:31.692922 13519 hybrid_clock.cc:648] HybridClock initialized: now 1765659571692871 us; error 41 us; skew 500 ppm
I20251213 20:59:31.693269 13519 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:31.695537 13519 webserver.cc:492] Webserver started at http://127.11.111.62:44033/ using document root <none> and password file <none>
I20251213 20:59:31.696350 13519 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:31.696468 13519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:31.701781 13519 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251213 20:59:31.704685 13534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:31.706068 13519 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251213 20:59:31.706220 13519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
uuid: "f2fb6eff9fdc4139b18baa7e29377276"
format_stamp: "Formatted at 2025-12-13 20:59:08 on dist-test-slave-p3t2"
I20251213 20:59:31.706907 13519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:31.747897 13519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:31.748766 13519 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:31.749019 13519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:31.768496 13585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.62:33723 every 8 connection(s)
I20251213 20:59:31.768498 13519 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.62:33723
I20251213 20:59:31.770457 13519 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-0/data/info.pb
I20251213 20:59:31.770916 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 13519
I20251213 20:59:31.777958 13586 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:31.781879 13586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Bootstrap starting.
I20251213 20:59:31.792709 13586 log.cc:826] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:31.820986 13586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=30 ignored=0} inserts{seen=23 ignored=0} mutations{seen=18 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251213 20:59:31.821709 13586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: Bootstrap complete.
I20251213 20:59:31.827143 13586 raft_consensus.cc:359] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:31.830125 13586 raft_consensus.cc:740] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fb6eff9fdc4139b18baa7e29377276, State: Initialized, Role: FOLLOWER
I20251213 20:59:31.830840 13586 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 5.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:31.831630 13591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:31.831881 13591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:31.832691 13586 sys_catalog.cc:565] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:31.845923 13605 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:31.846087 13605 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: loading cluster ID for follower catalog manager: success
I20251213 20:59:31.850680 13605 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: acquiring CA information for follower catalog manager: success
I20251213 20:59:31.854665 13605 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:31.873272 13519 master_runner.cc:186] Error getting master registration for 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111),
I20251213 20:59:31.878813 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:31.938771 12642 heartbeater.cc:344] Connected to a master server at 127.11.111.62:33723
I20251213 20:59:31.942590 13551 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:38789; Asking this server to re-register.
I20251213 20:59:31.943521 12642 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:31.945565 13551 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:32.128412 12362 heartbeater.cc:344] Connected to a master server at 127.11.111.62:33723
I20251213 20:59:32.130893 13551 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:44313; Asking this server to re-register.
I20251213 20:59:32.131570 12362 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:32.132920 13551 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:32.246187 13591 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:32.246541 13591 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:32.246840 12502 heartbeater.cc:344] Connected to a master server at 127.11.111.62:33723
I20251213 20:59:32.248615 13591 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:32.248704 13551 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:39923; Asking this server to re-register.
W20251213 20:59:32.249346 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:32.249279 13448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:32.249817 13448 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb6eff9fdc4139b18baa7e29377276 for earlier term 6. Current term is 7.
I20251213 20:59:32.250538 12502 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:32.250850 13538 leader_election.cc:400] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb6eff9fdc4139b18baa7e29377276 for earlier term 6. Current term is 7.
I20251213 20:59:32.250933 13538 leader_election.cc:403] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20251213 20:59:32.251111 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:32.251253 13591 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 5 FOLLOWER]: Advancing to term 7
I20251213 20:59:32.254899 13591 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb6eff9fdc4139b18baa7e29377276 for earlier term 6. Current term is 7.
I20251213 20:59:32.255942 13551 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:32.829720 13616 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:32.829998 13616 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:32.830839 13616 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:32.831205 13448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 8 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:32.831477 13448 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb6eff9fdc4139b18baa7e29377276 for term 8 because replica has last-logged OpId of term: 6 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 32.
W20251213 20:59:32.832751 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:32.832875 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:32.833209 13616 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:33.048483 13406 master_runner.cc:186] Error getting master registration for 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111),
I20251213 20:59:33.048691 13406 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:33.430687 13625 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:33.430891 13625 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:33.431506 13625 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:33.432087 13448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 8 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:33.432354 13448 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb6eff9fdc4139b18baa7e29377276 for term 8 because replica has last-logged OpId of term: 6 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 32.
W20251213 20:59:33.433486 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:33.433612 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:33.433859 13625 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:33.843905 11708 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskli_hB8/build/asan/bin/kudu pid 13406
I20251213 20:59:33.850836 13468 generic_service.cc:196] Checking for leaks (request via RPC)
I20251213 20:59:33.963429 11708 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskli_hB8/build/asan/bin/kudu with pid 13406
W20251213 20:59:33.980672 12651 connection.cc:537] client connection to 127.11.111.61:39203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:33.981572 13634 negotiation.cc:337] Failed RPC negotiation. Trace:
1213 20:59:33.980310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.111.61:39203 (local address 127.11.111.3:37427)
1213 20:59:33.980464 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1213 20:59:33.980481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1213 20:59:33.980592 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1213 20:59:33.981248 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":55}
I20251213 20:59:34.546101 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--webserver_interface=127.11.111.61
--webserver_port=42211
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251213 20:59:34.871227 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:34.871555 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:34.871662 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:34.881003 13637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:34.881125 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:34.881197 13637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:34.881255 13637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:34.891577 13637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.61:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.111.61
--webserver_port=42211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:34.893344 13637 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:34.895200 13637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:34.903275 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:34.904729 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:34.905694 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:34.906222 13637 server_base.cc:1047] running on GCE node
I20251213 20:59:34.907128 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:34.908663 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:34.909893 13637 hybrid_clock.cc:648] HybridClock initialized: now 1765659574909845 us; error 46 us; skew 500 ppm
I20251213 20:59:34.910267 13637 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:34.912647 13637 webserver.cc:492] Webserver started at http://127.11.111.61:42211/ using document root <none> and password file <none>
I20251213 20:59:34.913529 13637 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:34.913641 13637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:34.919744 13637 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251213 20:59:34.922890 13652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:34.924402 13637 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251213 20:59:34.924571 13637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
uuid: "68c0fc24cab047dcaa4cae2dbd628626"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:34.925293 13637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:34.952199 13637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:34.953078 13637 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:34.953344 13637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:34.972431 13637 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.61:39203
I20251213 20:59:34.972481 13703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.61:39203 every 8 connection(s)
I20251213 20:59:34.974097 13637 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-1/data/info.pb
I20251213 20:59:34.980930 13704 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:34.983203 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 13637
I20251213 20:59:34.985460 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap starting.
I20251213 20:59:34.998172 12643 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:34.998800 13704 log.cc:826] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:35.025501 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251213 20:59:35.026243 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Bootstrap complete.
I20251213 20:59:35.031517 13704 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:35.032298 13704 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Initialized, Role: FOLLOWER
I20251213 20:59:35.032900 13704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:35.033800 13709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:35.034073 13709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:35.034870 13704 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:35.038853 13719 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:35.039077 13719 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:35.039731 13719 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:35.044924 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:35.065351 13679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2fb6eff9fdc4139b18baa7e29377276" candidate_term: 8 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "68c0fc24cab047dcaa4cae2dbd628626" is_pre_election: true
I20251213 20:59:35.066210 13679 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb6eff9fdc4139b18baa7e29377276 for term 8 because replica has last-logged OpId of term: 6 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 32.
I20251213 20:59:35.067454 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:35.068608 13719 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:35.068682 13726 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:35.068748 13726 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: loading cluster ID for follower catalog manager: success
I20251213 20:59:35.072978 13726 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: acquiring CA information for follower catalog manager: success
I20251213 20:59:35.075906 13726 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:35.098563 13637 master_runner.cc:186] Error getting master registration for 127.11.111.60:36039: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111),
I20251213 20:59:35.102226 13637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:35.229038 12363 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:35.231017 13668 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:56133; Asking this server to re-register.
I20251213 20:59:35.231662 12363 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:35.233292 13668 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:35.308642 12503 heartbeater.cc:344] Connected to a master server at 127.11.111.61:39203
I20251213 20:59:35.310118 13668 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:48715; Asking this server to re-register.
I20251213 20:59:35.310736 12503 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:35.312178 13668 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:35.496351 13709 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:35.496636 13709 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:35.498433 13709 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:35.499060 13655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:35.499064 13561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 8 candidate_status { last_received { term: 6 index: 33 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
I20251213 20:59:35.499476 13561 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68c0fc24cab047dcaa4cae2dbd628626 in term 7.
I20251213 20:59:35.500016 13654 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276; no voters:
W20251213 20:59:35.500348 13655 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:35.500403 13709 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251213 20:59:35.500547 13709 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:35.500653 13709 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 7 FOLLOWER]: Advancing to term 8
I20251213 20:59:35.504428 13709 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:35.505419 13709 leader_election.cc:290] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
I20251213 20:59:35.505751 13561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68c0fc24cab047dcaa4cae2dbd628626" candidate_term: 8 candidate_status { last_received { term: 6 index: 33 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276"
I20251213 20:59:35.505954 13561 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 7 FOLLOWER]: Advancing to term 8
W20251213 20:59:35.507083 13655 leader_election.cc:336] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:35.507432 13668 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:56409; Asking this server to re-register.
I20251213 20:59:35.508044 12643 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:35.508566 13561 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68c0fc24cab047dcaa4cae2dbd628626 in term 8.
I20251213 20:59:35.508950 13654 leader_election.cc:304] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae
I20251213 20:59:35.509184 13709 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 8 FOLLOWER]: Leader election won for term 8
I20251213 20:59:35.509564 13709 raft_consensus.cc:697] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [term 8 LEADER]: Becoming Leader. State: Replica: 68c0fc24cab047dcaa4cae2dbd628626, State: Running, Role: LEADER
I20251213 20:59:35.509984 13709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:35.512048 13668 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:35.512173 13733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c0fc24cab047dcaa4cae2dbd628626. Latest consensus state: current_term: 8 leader_uuid: "68c0fc24cab047dcaa4cae2dbd628626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:35.512439 13733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626 [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:35.513192 13735 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:35.515511 13735 catalog_manager.cc:679] Loaded metadata for table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]
I20251213 20:59:35.515906 13735 catalog_manager.cc:679] Loaded metadata for table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]
I20251213 20:59:35.516211 13735 catalog_manager.cc:679] Loaded metadata for table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]
I20251213 20:59:35.516505 13735 catalog_manager.cc:679] Loaded metadata for table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]
I20251213 20:59:35.516798 13735 catalog_manager.cc:679] Loaded metadata for table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]
I20251213 20:59:35.517087 13735 catalog_manager.cc:679] Loaded metadata for table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]
I20251213 20:59:35.517359 13735 catalog_manager.cc:679] Loaded metadata for table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]
I20251213 20:59:35.517711 13735 catalog_manager.cc:679] Loaded metadata for table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]
I20251213 20:59:35.517982 13735 catalog_manager.cc:679] Loaded metadata for table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]
I20251213 20:59:35.519744 13735 tablet_loader.cc:96] loaded metadata for tablet 529995cf7f2e4b15b5e026178d59a916 (table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261])
I20251213 20:59:35.520190 13735 tablet_loader.cc:96] loaded metadata for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5])
I20251213 20:59:35.520629 13735 tablet_loader.cc:96] loaded metadata for tablet cb1f2d25f95c49b4918383371cc12368 (table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3])
I20251213 20:59:35.520889 13735 tablet_loader.cc:96] loaded metadata for tablet cd58b344007c431ab5d8771d201733b7 (table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3])
I20251213 20:59:35.521123 13735 tablet_loader.cc:96] loaded metadata for tablet d03a7cadb5a044a99f6f18ca2ee4fb36 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765])
I20251213 20:59:35.521399 13735 tablet_loader.cc:96] loaded metadata for tablet da4c8619ce49439cb7b84eb17d587fb4 (table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60])
I20251213 20:59:35.521690 13735 tablet_loader.cc:96] loaded metadata for tablet f61276e680a24502bd0d0547a55b2abd (table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72])
I20251213 20:59:35.521948 13735 tablet_loader.cc:96] loaded metadata for tablet fa028486f240481f9081ea514153e384 (table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18])
I20251213 20:59:35.522208 13735 tablet_loader.cc:96] loaded metadata for tablet ff555a43366144fdb7f5c857d351ef3b (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153])
I20251213 20:59:35.522425 13735 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:35.523310 13735 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:35.523373 13735 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:35.524505 13735 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:35.525321 13735 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 68c0fc24cab047dcaa4cae2dbd628626: Loaded TSK: 0
I20251213 20:59:35.526156 13735 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:35.526966 13735 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:35.527745 13735 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:35.653662 13669 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32890:
table { table_name: "default.Table_4ece212e4b3b44968882ae1e8ca68422" } modify_external_catalogs: true
I20251213 20:59:35.653621 13668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32890:
name: "default.Table_60f4a42abdd34e9fbfa1e25215d9fba5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:35.653991 13669 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32890:
table { table_name: "default.Table_4ece212e4b3b44968882ae1e8ca68422" } modify_external_catalogs: true
I20251213 20:59:35.654373 13663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32890:
name: "default.Table_32850fadebae44e797c016b5161038ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:35.654800 13667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32890:
name: "default.Table_cc9c8f278dee46108580b4455e30b213"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:35.654966 13664 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32890:
table { table_name: "default.Table_12be436a89df4cfcac0b036dbab2d770" } modify_external_catalogs: true
I20251213 20:59:35.655200 13664 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32890:
table { table_name: "default.Table_12be436a89df4cfcac0b036dbab2d770" } modify_external_catalogs: true
I20251213 20:59:35.655687 13665 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32890:
name: "default.Table_29cded03ad224e8d809f516d66e903dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:35.655661 13666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32890:
name: "default.Table_9857f31a4d1e42b78febc6e2e68587fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:35.655932 13662 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32890:
table { table_name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb" } modify_external_catalogs: true
I20251213 20:59:35.656210 13662 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32890:
table { table_name: "default.Table_ceb9db759caa483dbbe2ac726dc7f8bb" } modify_external_catalogs: true
I20251213 20:59:35.656463 13660 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32890:
name: "default.Table_eb562e37a81541048c6a9757feca4265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:35.656757 13661 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:32890:
table { table_name: "default.Table_b7f4c624bfff43c8a5177f45a23ff719" } new_table_name: "default.Table_cbfd66dd739f453f8b64a7f6eaf0b21b" modify_external_catalogs: true
F20251213 20:59:35.660646 13725 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_ebe9a0444b8a40c7ac9d635be30c28b0, message={"server":"","servicePrincipal":"","db":"default","table":"table_ebe9a0444b8a40c7ac9d635be30c28b0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ebe9a0444b8a40c7ac9d635be30c28b0\",\"kudu.table_id\":\"2d6fd96071954bcb92508900e2a386b6\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_b027752248784b2f9fff90508e2b62b4, message={"server":"","servicePrincipal":"","db":"default","table":"table_b027752248784b2f9fff90508e2b62b4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b027752248784b2f9fff90508e2b62b4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_b027752248784b2f9fff90508e2b62b4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b027752248784b2f9fff90508e2b62b4\",\"kudu.table_id\":\"abe1a66f87914e24b17972fa42e24765\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f1e2920cdcd google::LogMessage::Fail() at ??:0
@ 0x7f1e29210b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1e2920c7cc google::LogMessage::Flush() at ??:0
@ 0x7f1e2920ca11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f1e3fbf2aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f1e3fbf1351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f1e3fbf8fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f1e3fbf8e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f1e2bf813f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1e2f58c6db start_thread at ??:0
@ 0x7f1e280bd71f clone at ??:0
W20251213 20:59:35.848273 12392 connection.cc:537] client connection to 127.11.111.61:39203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:35.848306 12252 connection.cc:537] client connection to 127.11.111.61:39203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:35.850028 12651 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.111.61:39203)
W20251213 20:59:35.850628 12651 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.111.61:39203)
W20251213 20:59:35.850848 12651 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.111.61:39203)
W20251213 20:59:35.851083 12651 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.111.61:39203)
I20251213 20:59:36.881747 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:36.881927 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:37.588814 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--webserver_interface=127.11.111.60
--webserver_port=36499
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251213 20:59:37.671038 13756 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:37.671264 13756 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:37.671973 13756 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:37.672928 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251213 20:59:37.674039 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:37.674404 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:37.674527 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:37.674783 13756 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251213 20:59:37.926077 13755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:37.926389 13755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:37.926467 13755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:37.934906 13755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:37.935017 13755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:37.935081 13755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:37.935127 13755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:37.944963 13755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.111.60
--webserver_port=36499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:37.946597 13755 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:37.948518 13755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:37.956116 13765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:37.956647 13764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:37.956957 13767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:37.957185 13755 server_base.cc:1047] running on GCE node
I20251213 20:59:37.957971 13755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:37.959362 13755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:37.960563 13755 hybrid_clock.cc:648] HybridClock initialized: now 1765659577960511 us; error 48 us; skew 500 ppm
I20251213 20:59:37.960881 13755 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:37.963042 13755 webserver.cc:492] Webserver started at http://127.11.111.60:36499/ using document root <none> and password file <none>
I20251213 20:59:37.963785 13755 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:37.963888 13755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:37.969409 13755 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20251213 20:59:37.972888 13773 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:37.974344 13755 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251213 20:59:37.974519 13755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
uuid: "6736d1c95f384b9098c089a20b76e9ae"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:37.975193 13755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:38.019888 13755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:38.020766 13755 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:38.021026 13755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:38.040690 13755 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.60:36039
I20251213 20:59:38.040827 13824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.60:36039 every 8 connection(s)
I20251213 20:59:38.042346 13755 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
I20251213 20:59:38.047060 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 13755
I20251213 20:59:38.050714 13825 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:38.054462 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap starting.
I20251213 20:59:38.066597 13825 log.cc:826] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:38.092572 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=1 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251213 20:59:38.093215 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap complete.
I20251213 20:59:38.098485 13825 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:38.099416 13825 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Initialized, Role: FOLLOWER
I20251213 20:59:38.100107 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 7.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:38.100797 13829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:38.101040 13829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:38.101845 13825 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:38.122947 13843 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:38.123128 13843 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: loading cluster ID for follower catalog manager: success
I20251213 20:59:38.127334 13843 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: acquiring CA information for follower catalog manager: success
I20251213 20:59:38.130889 13843 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:38.140859 13755 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:38.144379 13755 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:38.292618 12501 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:38.293886 12361 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:38.295533 13790 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:59437; Asking this server to re-register.
I20251213 20:59:38.295547 13789 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:51479; Asking this server to re-register.
I20251213 20:59:38.296259 12501 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:38.296353 12361 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:38.298110 13790 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:38.298233 13789 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:38.461777 12641 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:38.463359 13789 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:52709; Asking this server to re-register.
I20251213 20:59:38.464463 12641 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:38.465840 13790 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:38.681806 13850 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:38.682063 13850 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:38.684005 13850 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:38.684602 13561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 8 candidate_status { last_received { term: 7 index: 34 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
I20251213 20:59:38.684947 13561 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6736d1c95f384b9098c089a20b76e9ae in current term 8: Already voted for candidate 68c0fc24cab047dcaa4cae2dbd628626 in this term.
W20251213 20:59:38.685917 13777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:38.687278 13777 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:38.687460 13777 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae; no voters: 68c0fc24cab047dcaa4cae2dbd628626, f2fb6eff9fdc4139b18baa7e29377276
I20251213 20:59:38.687799 13850 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 7 FOLLOWER]: Advancing to term 8
I20251213 20:59:38.691362 13850 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251213 20:59:39.296898 13857 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:39.297083 13857 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:39.297842 13857 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:39.298410 13561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 9 candidate_status { last_received { term: 7 index: 34 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
I20251213 20:59:39.298650 13561 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 8.
I20251213 20:59:39.299134 13775 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:39.299477 13857 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251213 20:59:39.299626 13857 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:39.299722 13857 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 8 FOLLOWER]: Advancing to term 9
W20251213 20:59:39.300102 13777 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:39.302101 13857 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:39.302868 13857 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 9 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:39.303334 13561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 9 candidate_status { last_received { term: 7 index: 34 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276"
I20251213 20:59:39.303576 13561 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 8 FOLLOWER]: Advancing to term 9
W20251213 20:59:39.304368 13777 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:39.305701 13561 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 9.
I20251213 20:59:39.306100 13775 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters: 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:39.306413 13857 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Leader election won for term 9
I20251213 20:59:39.306777 13857 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 LEADER]: Becoming Leader. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Running, Role: LEADER
I20251213 20:59:39.307215 13857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 7.34, Last appended by leader: 34, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:39.309919 13860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 9 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:39.310190 13860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:39.310860 13862 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:39.313362 13862 catalog_manager.cc:679] Loaded metadata for table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]
I20251213 20:59:39.313807 13862 catalog_manager.cc:679] Loaded metadata for table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]
I20251213 20:59:39.314106 13862 catalog_manager.cc:679] Loaded metadata for table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]
I20251213 20:59:39.314390 13862 catalog_manager.cc:679] Loaded metadata for table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]
I20251213 20:59:39.314648 13862 catalog_manager.cc:679] Loaded metadata for table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]
I20251213 20:59:39.314917 13862 catalog_manager.cc:679] Loaded metadata for table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]
I20251213 20:59:39.315181 13862 catalog_manager.cc:679] Loaded metadata for table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]
I20251213 20:59:39.315459 13862 catalog_manager.cc:679] Loaded metadata for table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]
I20251213 20:59:39.315726 13862 catalog_manager.cc:679] Loaded metadata for table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]
I20251213 20:59:39.317566 13862 tablet_loader.cc:96] loaded metadata for tablet 529995cf7f2e4b15b5e026178d59a916 (table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261])
I20251213 20:59:39.318006 13862 tablet_loader.cc:96] loaded metadata for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5])
I20251213 20:59:39.318408 13862 tablet_loader.cc:96] loaded metadata for tablet cb1f2d25f95c49b4918383371cc12368 (table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3])
I20251213 20:59:39.318655 13862 tablet_loader.cc:96] loaded metadata for tablet cd58b344007c431ab5d8771d201733b7 (table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3])
I20251213 20:59:39.318859 13862 tablet_loader.cc:96] loaded metadata for tablet d03a7cadb5a044a99f6f18ca2ee4fb36 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765])
I20251213 20:59:39.319072 13862 tablet_loader.cc:96] loaded metadata for tablet da4c8619ce49439cb7b84eb17d587fb4 (table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60])
I20251213 20:59:39.319293 13862 tablet_loader.cc:96] loaded metadata for tablet f61276e680a24502bd0d0547a55b2abd (table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72])
I20251213 20:59:39.319527 13862 tablet_loader.cc:96] loaded metadata for tablet fa028486f240481f9081ea514153e384 (table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18])
I20251213 20:59:39.319761 13862 tablet_loader.cc:96] loaded metadata for tablet ff555a43366144fdb7f5c857d351ef3b (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153])
I20251213 20:59:39.319967 13862 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:39.320879 13862 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:39.320947 13862 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:39.322198 13862 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:39.323020 13862 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Loaded TSK: 0
I20251213 20:59:39.323913 13862 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:39.325172 13862 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:39.326185 13862 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:39.412686 13790 master_service.cc:954] ReplaceTablet: received request to replace tablet f61276e680a24502bd0d0547a55b2abd from {username='slave'} at 127.0.0.1:44280
I20251213 20:59:39.412827 13789 master_service.cc:954] ReplaceTablet: received request to replace tablet 529995cf7f2e4b15b5e026178d59a916 from {username='slave'} at 127.0.0.1:44280
I20251213 20:59:39.413250 13788 master_service.cc:954] ReplaceTablet: received request to replace tablet da4c8619ce49439cb7b84eb17d587fb4 from {username='slave'} at 127.0.0.1:44280
I20251213 20:59:39.413338 13786 master_service.cc:954] ReplaceTablet: received request to replace tablet da4c8619ce49439cb7b84eb17d587fb4 from {username='slave'} at 127.0.0.1:44280
I20251213 20:59:39.414057 13784 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44280:
name: "default.Table_0667e63914504edaa41a24b9c799bfba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:39.415328 13787 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d2e8c74b4ca427ab675b574516c1db1 from {username='slave'} at 127.0.0.1:44280
I20251213 20:59:39.416134 13787 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44280:
name: "default.Table_9857f31a4d1e42b78febc6e2e68587fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:39.416849 13785 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44280:
table { table_name: "default.Table_e15c8fc8dfd143aaa4b8d039e22cf63d" } new_table_name: "default.Table_40016a174bfc4e3c9f29ec786589b626" modify_external_catalogs: true
I20251213 20:59:39.417780 13782 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44280:
table { table_name: "default.Table_12be436a89df4cfcac0b036dbab2d770" } modify_external_catalogs: true
I20251213 20:59:39.418056 13782 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44280:
table { table_name: "default.Table_12be436a89df4cfcac0b036dbab2d770" } modify_external_catalogs: true
I20251213 20:59:39.417997 13781 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44280:
name: "default.Table_29cded03ad224e8d809f516d66e903dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:39.418299 13783 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44280:
table { table_name: "default.Table_1368482eba894f4c8b68346432f132d8" } new_table_name: "default.Table_3c93b9a42ca143889e8ab38b6fc45d47" modify_external_catalogs: true
I20251213 20:59:39.420850 13561 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Refusing update from remote peer 6736d1c95f384b9098c089a20b76e9ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 9 index: 36. (index mismatch)
I20251213 20:59:39.421677 13860 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 30, Time since last communication: 0.000s
W20251213 20:59:39.422434 13777 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae -> Peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Couldn't send request to peer 68c0fc24cab047dcaa4cae2dbd628626. Status: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
F20251213 20:59:39.423569 13842 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_ebe9a0444b8a40c7ac9d635be30c28b0, message={"server":"","servicePrincipal":"","db":"default","table":"table_ebe9a0444b8a40c7ac9d635be30c28b0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ebe9a0444b8a40c7ac9d635be30c28b0\",\"kudu.table_id\":\"2d6fd96071954bcb92508900e2a386b6\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_b027752248784b2f9fff90508e2b62b4, message={"server":"","servicePrincipal":"","db":"default","table":"table_b027752248784b2f9fff90508e2b62b4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b027752248784b2f9fff90508e2b62b4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_b027752248784b2f9fff90508e2b62b4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b027752248784b2f9fff90508e2b62b4\",\"kudu.table_id\":\"abe1a66f87914e24b17972fa42e24765\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f379dc6fdcd google::LogMessage::Fail() at ??:0
@ 0x7f379dc73b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f379dc6f7cc google::LogMessage::Flush() at ??:0
@ 0x7f379dc6fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f37b4655aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f37b4654351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f37b465bfcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f37b465be49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7f37a09e43f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f37a3fef6db start_thread at ??:0
@ 0x7f379cb2071f clone at ??:0
I20251213 20:59:39.462107 13872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:39.462389 13872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:39.462218 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 9 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:39.462519 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:39.462819 13878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:39.463049 13878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
W20251213 20:59:39.617417 12531 connection.cc:537] client connection to 127.11.111.60:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:39.617465 13536 connection.cc:537] server connection from 127.0.0.1:39686 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:39.618306 12650 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:39.618690 12650 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:39.619051 12650 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:39.619468 12650 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.111.60:36039)
I20251213 20:59:39.924901 13878 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:39.925143 13878 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:39.926033 13878 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:39.928385 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:39.928588 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:39.928704 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:39.928951 13878 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251213 20:59:40.528702 13891 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:40.528894 13891 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:40.529582 13891 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:40.531302 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:40.531462 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:40.531586 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:40.531795 13891 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251213 20:59:41.014458 11708 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskli_hB8/build/asan/bin/kudu
/tmp/dist-test-taskli_hB8/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--webserver_interface=127.11.111.60
--webserver_port=36499
--builtin_ntp_servers=127.11.111.20:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251213 20:59:41.185909 13896 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:41.186121 13896 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:41.186770 13896 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:41.189033 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:41.189219 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:41.189363 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:41.189673 13896 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251213 20:59:41.345497 13895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251213 20:59:41.345829 13895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251213 20:59:41.345933 13895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251213 20:59:41.354218 13895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251213 20:59:41.354324 13895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251213 20:59:41.354404 13895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251213 20:59:41.354463 13895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251213 20:59:41.364828 13895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.111.20:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45019
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.111.60:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.111.60
--webserver_port=36499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 13 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9500
ASAN enabled
I20251213 20:59:41.366571 13895 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251213 20:59:41.368393 13895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251213 20:59:41.377519 13905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:41.377492 13906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251213 20:59:41.377692 13908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251213 20:59:41.378851 13895 server_base.cc:1047] running on GCE node
I20251213 20:59:41.379616 13895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251213 20:59:41.381071 13895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251213 20:59:41.382282 13895 hybrid_clock.cc:648] HybridClock initialized: now 1765659581382206 us; error 64 us; skew 500 ppm
I20251213 20:59:41.382665 13895 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251213 20:59:41.384895 13895 webserver.cc:492] Webserver started at http://127.11.111.60:36499/ using document root <none> and password file <none>
I20251213 20:59:41.385682 13895 fs_manager.cc:362] Metadata directory not provided
I20251213 20:59:41.385771 13895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251213 20:59:41.391153 13895 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20251213 20:59:41.394361 13914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251213 20:59:41.395964 13895 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251213 20:59:41.396122 13895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data,/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
uuid: "6736d1c95f384b9098c089a20b76e9ae"
format_stamp: "Formatted at 2025-12-13 20:59:09 on dist-test-slave-p3t2"
I20251213 20:59:41.396832 13895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251213 20:59:41.421495 13895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251213 20:59:41.422412 13895 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251213 20:59:41.422686 13895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251213 20:59:41.442780 13895 rpc_server.cc:307] RPC server started. Bound to: 127.11.111.60:36039
I20251213 20:59:41.442800 13965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.111.60:36039 every 8 connection(s)
I20251213 20:59:41.444429 13895 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/master-2/data/info.pb
I20251213 20:59:41.451432 13966 sys_catalog.cc:263] Verifying existing consensus state
I20251213 20:59:41.452030 11708 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskli_hB8/build/asan/bin/kudu as pid 13895
I20251213 20:59:41.455296 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap starting.
I20251213 20:59:41.467154 13966 log.cc:826] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Log is configured to *not* fsync() on all Append() calls
I20251213 20:59:41.493738 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=1 applied=33 ignored=0} inserts{seen=24 ignored=0} mutations{seen=19 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20251213 20:59:41.494405 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Bootstrap complete.
I20251213 20:59:41.501791 13966 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:41.504881 13966 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Initialized, Role: FOLLOWER
I20251213 20:59:41.505568 13966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 9.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:41.506326 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:41.506604 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:41.507520 13966 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: configured and running, proceeding with master startup.
I20251213 20:59:41.521219 13985 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:41.521462 13985 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: loading cluster ID for follower catalog manager: success
I20251213 20:59:41.525978 13985 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: acquiring CA information for follower catalog manager: success
I20251213 20:59:41.529031 13985 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251213 20:59:41.544586 13895 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:41.550062 13895 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:41.630468 12641 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:41.632941 13931 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c5976941698547af85c98dff50a9d45b" instance_seqno: 1765659551586488) as {username='slave'} at 127.11.111.3:53721; Asking this server to re-register.
I20251213 20:59:41.633713 12641 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:41.635422 13931 ts_manager.cc:194] Registered new tserver with Master: c5976941698547af85c98dff50a9d45b (127.11.111.3:32819)
I20251213 20:59:41.817289 12501 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:41.817675 12361 heartbeater.cc:344] Connected to a master server at 127.11.111.60:36039
I20251213 20:59:41.818893 13931 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1b75487eefe54cbdb4140dea8ab8a499" instance_seqno: 1765659551098490) as {username='slave'} at 127.11.111.2:60347; Asking this server to re-register.
I20251213 20:59:41.819203 13930 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "97f3d76e88294646bccacef46bb93c38" instance_seqno: 1765659550562892) as {username='slave'} at 127.11.111.1:44295; Asking this server to re-register.
I20251213 20:59:41.819458 12501 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:41.819808 12361 heartbeater.cc:461] Registering TS with master...
I20251213 20:59:41.820889 13930 ts_manager.cc:194] Registered new tserver with Master: 1b75487eefe54cbdb4140dea8ab8a499 (127.11.111.2:40023)
I20251213 20:59:41.821223 13931 ts_manager.cc:194] Registered new tserver with Master: 97f3d76e88294646bccacef46bb93c38 (127.11.111.1:36681)
I20251213 20:59:41.884624 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:41.884802 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:42.029331 13993 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251213 20:59:42.029654 13993 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:42.031761 13993 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:42.032509 13561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 10 candidate_status { last_received { term: 9 index: 38 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276" is_pre_election: true
W20251213 20:59:42.032675 13918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:42.032778 13561 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 9.
I20251213 20:59:42.033375 13916 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters:
I20251213 20:59:42.033802 13993 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251213 20:59:42.034101 13993 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251213 20:59:42.034226 13993 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 9 FOLLOWER]: Advancing to term 10
W20251213 20:59:42.034255 13918 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:42.037606 13993 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:42.038252 13993 leader_election.cc:290] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 10 election: Requested vote from peers f2fb6eff9fdc4139b18baa7e29377276 (127.11.111.62:33723), 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203)
I20251213 20:59:42.038708 13561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6736d1c95f384b9098c089a20b76e9ae" candidate_term: 10 candidate_status { last_received { term: 9 index: 38 } } ignore_live_leader: false dest_uuid: "f2fb6eff9fdc4139b18baa7e29377276"
I20251213 20:59:42.038951 13561 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 9 FOLLOWER]: Advancing to term 10
W20251213 20:59:42.039579 13918 leader_election.cc:336] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:42.041043 13561 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6736d1c95f384b9098c089a20b76e9ae in term 10.
I20251213 20:59:42.041496 13916 leader_election.cc:304] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6736d1c95f384b9098c089a20b76e9ae, f2fb6eff9fdc4139b18baa7e29377276; no voters: 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:42.041837 13993 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 10 FOLLOWER]: Leader election won for term 10
I20251213 20:59:42.042258 13993 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [term 10 LEADER]: Becoming Leader. State: Replica: 6736d1c95f384b9098c089a20b76e9ae, State: Running, Role: LEADER
I20251213 20:59:42.042733 13993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 9.38, Last appended by leader: 38, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:42.045360 13995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.045656 13995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:42.499572 13561 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Refusing update from remote peer 6736d1c95f384b9098c089a20b76e9ae: Log matching property violated. Preceding OpId in replica: term: 9 index: 38. Preceding OpId from leader: term: 10 index: 39. (index mismatch)
I20251213 20:59:42.500308 13995 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 33, Time since last communication: 0.000s
I20251213 20:59:42.502394 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6736d1c95f384b9098c089a20b76e9ae. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.502667 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:42.510504 13995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.510686 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.510834 13995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:42.510915 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:42.510605 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.511094 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae [sys.catalog]: This master's current role is: LEADER
I20251213 20:59:42.511693 14013 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.511823 14006 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.512141 14006 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:42.512008 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "6736d1c95f384b9098c089a20b76e9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } } }
I20251213 20:59:42.512238 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
I20251213 20:59:42.512012 14013 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [sys.catalog]: This master's current role is: FOLLOWER
W20251213 20:59:42.512622 13918 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae -> Peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Couldn't send request to peer 68c0fc24cab047dcaa4cae2dbd628626. Status: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251213 20:59:42.562672 13997 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251213 20:59:42.565508 13997 catalog_manager.cc:679] Loaded metadata for table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3]
I20251213 20:59:42.566017 13997 catalog_manager.cc:679] Loaded metadata for table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18]
I20251213 20:59:42.566365 13997 catalog_manager.cc:679] Loaded metadata for table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765]
I20251213 20:59:42.566681 13997 catalog_manager.cc:679] Loaded metadata for table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261]
I20251213 20:59:42.566982 13997 catalog_manager.cc:679] Loaded metadata for table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153]
I20251213 20:59:42.567312 13997 catalog_manager.cc:679] Loaded metadata for table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5]
I20251213 20:59:42.567612 13997 catalog_manager.cc:679] Loaded metadata for table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72]
I20251213 20:59:42.567926 13997 catalog_manager.cc:679] Loaded metadata for table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60]
I20251213 20:59:42.568215 13997 catalog_manager.cc:679] Loaded metadata for table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3]
I20251213 20:59:42.569976 13997 tablet_loader.cc:96] loaded metadata for tablet 108303fedb0e47f3a30e91b4d8ccfcd4 (table default.table_ceb9db759caa483dbbe2ac726dc7f8bb [id=eec24cb8c3c142f48350b073c99e4d60])
I20251213 20:59:42.570503 13997 tablet_loader.cc:96] loaded metadata for tablet 16958ecfb87e420bbcf85b1289a19db3 (table default.table_12be436a89df4cfcac0b036dbab2d770 [id=e211037a5d9045ada23784889d3fdb72])
I20251213 20:59:42.570960 13997 tablet_loader.cc:96] loaded metadata for tablet 6fbc589dffdd4ce38c15801fa128fc2a (table default.table_e15c8fc8dfd143aaa4b8d039e22cf63d [id=c299c8ffc6a7470691ac95ab24cd6261])
I20251213 20:59:42.571381 13997 tablet_loader.cc:96] loaded metadata for tablet 7f43dfbead524dc6aa1cddfbafe6e639 (table default.table_e09d8c7259974a359a976428a2b351bb [id=e09c4151ad45495896b7cc754c5834c5])
I20251213 20:59:42.571816 13997 tablet_loader.cc:96] loaded metadata for tablet cb1f2d25f95c49b4918383371cc12368 (table default.table_1368482eba894f4c8b68346432f132d8 [id=3defb3020c584a7587e3632511396bd3])
I20251213 20:59:42.572068 13997 tablet_loader.cc:96] loaded metadata for tablet cd58b344007c431ab5d8771d201733b7 (table default.table_b7f4c624bfff43c8a5177f45a23ff719 [id=f78dc8504fd04c5e8dc48276891a7ba3])
I20251213 20:59:42.572307 13997 tablet_loader.cc:96] loaded metadata for tablet d03a7cadb5a044a99f6f18ca2ee4fb36 (table default.table_b027752248784b2f9fff90508e2b62b4 [id=abe1a66f87914e24b17972fa42e24765])
I20251213 20:59:42.572860 13997 tablet_loader.cc:96] loaded metadata for tablet fa028486f240481f9081ea514153e384 (table default.table_4ece212e4b3b44968882ae1e8ca68422 [id=62f666dd15494a0eb095b9f9ded82d18])
I20251213 20:59:42.573132 13997 tablet_loader.cc:96] loaded metadata for tablet ff555a43366144fdb7f5c857d351ef3b (table default.table_0dc454009abd48db8c91e0aa3c76eb4c [id=d2519a98a46e45618d21e90dd0348153])
I20251213 20:59:42.573405 13997 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251213 20:59:42.574443 13997 catalog_manager.cc:1269] Loaded cluster ID: 4c02c184008342ac9b3f936f1e5b7943
I20251213 20:59:42.574542 13997 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251213 20:59:42.575999 13997 catalog_manager.cc:1514] Loading token signing keys...
I20251213 20:59:42.577030 13997 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 6736d1c95f384b9098c089a20b76e9ae: Loaded TSK: 0
I20251213 20:59:42.577989 13997 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251213 20:59:42.578928 13997 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20251213 20:59:42.579816 13997 catalog_manager.cc:5876] Last processed Hive Metastore notification event ID: -1
I20251213 20:59:42.621219 13930 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36566:
table { table_name: "default.Table_b027752248784b2f9fff90508e2b62b4" } new_table_name: "default.Table_31c612ce45a347a2b23e54b30cf823be" modify_external_catalogs: true
I20251213 20:59:42.621739 13926 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36566:
table { table_name: "default.Table_e09d8c7259974a359a976428a2b351bb" } modify_external_catalogs: true
I20251213 20:59:42.621240 13931 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36566:
table { table_name: "default.Table_0dc454009abd48db8c91e0aa3c76eb4c" } new_table_name: "default.Table_7c733924e7c54175b89a1b37e0095c4b" modify_external_catalogs: true
I20251213 20:59:42.621991 13926 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36566:
table { table_name: "default.Table_e09d8c7259974a359a976428a2b351bb" } modify_external_catalogs: true
I20251213 20:59:42.621893 13929 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36566:
name: "default.Table_5009d0ba1a394917a7101259634328ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:42.622205 13927 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36566:
name: "default.Table_e448c113874f43c6b7a38f4b918a08a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:42.623059 13928 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36566:
name: "default.Table_9e21ceee05264206866e8b0d62a537c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:42.623255 13923 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36566:
table { table_name: "default.Table_e15c8fc8dfd143aaa4b8d039e22cf63d" } new_table_name: "default.Table_40016a174bfc4e3c9f29ec786589b626" modify_external_catalogs: true
I20251213 20:59:42.623531 13924 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36566:
table { table_name: "default.Table_b7f4c624bfff43c8a5177f45a23ff719" } new_table_name: "default.Table_cbfd66dd739f453f8b64a7f6eaf0b21b" modify_external_catalogs: true
I20251213 20:59:42.624012 13922 master_service.cc:954] ReplaceTablet: received request to replace tablet da4c8619ce49439cb7b84eb17d587fb4 from {username='slave'} at 127.0.0.1:36566
I20251213 20:59:42.624061 13925 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36566:
name: "default.Table_eb562e37a81541048c6a9757feca4265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251213 20:59:42.625016 13922 master_service.cc:954] ReplaceTablet: received request to replace tablet da4c8619ce49439cb7b84eb17d587fb4 from {username='slave'} at 127.0.0.1:36566
I20251213 20:59:42.625456 13922 master_service.cc:954] ReplaceTablet: received request to replace tablet f61276e680a24502bd0d0547a55b2abd from {username='slave'} at 127.0.0.1:36566
I20251213 20:59:42.625844 13922 master_service.cc:954] ReplaceTablet: received request to replace tablet 529995cf7f2e4b15b5e026178d59a916 from {username='slave'} at 127.0.0.1:36566
I20251213 20:59:42.626312 13922 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36566:
name: "default.Table_0667e63914504edaa41a24b9c799bfba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20251213 20:59:42.628137 13984 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_ebe9a0444b8a40c7ac9d635be30c28b0, message={"server":"","servicePrincipal":"","db":"default","table":"table_ebe9a0444b8a40c7ac9d635be30c28b0","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_ebe9a0444b8a40c7ac9d635be30c28b0\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_ebe9a0444b8a40c7ac9d635be30c28b0\",\"kudu.table_id\":\"2d6fd96071954bcb92508900e2a386b6\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1765659552, eventType=CREATE_TABLE, dbName=default, tableName=table_b027752248784b2f9fff90508e2b62b4, message={"server":"","servicePrincipal":"","db":"default","table":"table_b027752248784b2f9fff90508e2b62b4","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_b027752248784b2f9fff90508e2b62b4\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1765659551},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskli_hB8/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1765659532758167-11708-0/minicluster-data/warehouse/table_b027752248784b2f9fff90508e2b62b4\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_b027752248784b2f9fff90508e2b62b4\",\"kudu.table_id\":\"abe1a66f87914e24b17972fa42e24765\",\"transient_lastDdlTime\":\"1765659551\",\"comment\":\"\",\"kudu.master_addresses\":\"127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"4c02c184008342ac9b3f936f1e5b7943\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1765659552,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7ff5f0e0ddcd google::LogMessage::Fail() at ??:0
@ 0x7ff5f0e11b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff5f0e0d7cc google::LogMessage::Flush() at ??:0
@ 0x7ff5f0e0da11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ff6077f3aa4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7ff6077f2351 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7ff6077f9fcc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7ff6077f9e49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8970d std::function<>::operator()() at ??:?
@ 0x7ff5f3b823f9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff5f718d6db start_thread at ??:0
@ 0x7ff5efcbe71f clone at ??:0
W20251213 20:59:42.831825 12391 connection.cc:537] client connection to 127.11.111.60:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:42.831856 12251 connection.cc:537] client connection to 127.11.111.60:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:42.832463 12650 connection.cc:537] client connection to 127.11.111.60:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251213 20:59:42.833117 12650 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.11.111.60:36039)
W20251213 20:59:42.833750 12650 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:42.833946 12650 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.111.60:36039)
W20251213 20:59:42.834865 12650 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.11.111.60:36039)
I20251213 20:59:43.004361 14013 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:43.004590 14013 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:43.005261 14013 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:43.006358 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20251213 20:59:43.007578 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:43.007768 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:43.007860 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:43.008069 14013 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 20:59:43.660198 14034 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:43.660416 14034 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:43.661114 14034 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:43.663066 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:43.663254 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:43.663353 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:43.663580 14034 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 20:59:43.962775 11708 master-stress-test.cc:577] Killing test thread 1/25
I20251213 20:59:44.332546 14039 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:44.332772 14039 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:44.333575 14039 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:44.336184 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:44.336401 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:44.336516 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:44.336733 14039 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 20:59:45.118954 14044 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:45.119166 14044 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:45.119843 14044 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:45.121856 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:45.122025 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:45.122150 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:45.122407 14044 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 20:59:46.762857 14050 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:46.763070 14050 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:46.763667 14050 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:46.765808 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:46.765959 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:46.766121 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:46.766495 14050 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 20:59:46.887638 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:46.887835 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:50.168999 14056 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:50.169312 14056 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:50.170055 14056 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:50.174235 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251213 20:59:50.175647 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 20:59:50.175846 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 20:59:50.175949 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:50.176198 14056 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 20:59:51.890473 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:51.890658 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:54.829499 14066 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:54.829741 14066 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:54.830411 14066 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:54.832183 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:54.832333 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:54.832473 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:54.832701 14066 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 20:59:56.893805 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 20:59:56.894008 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 20:59:58.449944 14076 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 20:59:58.450157 14076 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 20:59:58.450891 14076 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 20:59:58.451886 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251213 20:59:58.453150 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 20:59:58.453325 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 20:59:58.453492 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 20:59:58.453732 14076 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:00:01.896790 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:01.896979 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:06.900264 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:06.900511 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:11.201942 14097 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:00:11.202219 14097 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:00:11.202956 14097 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:00:11.203859 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:00:11.205088 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:00:11.205286 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:00:11.205425 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:00:11.205689 14097 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:00:11.904052 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:11.904238 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251213 21:00:12.404824 12641 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (59 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:00:12.404912 12501 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (59 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:00:12.404946 12361 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (59 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:00:12.439343 14104 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:00:12.439513 14104 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:00:12.440114 14104 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:00:12.441828 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:00:12.441998 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:00:12.442106 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:00:12.442329 14104 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:00:16.907086 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:16.907275 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:21.910612 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:21.910810 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:26.914379 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:26.914601 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:29.823167 14136 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:00:29.823376 14136 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:00:29.824232 14136 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:00:29.825192 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251213 21:00:29.826704 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:00:29.826897 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:00:29.827032 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:00:29.827314 14136 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:00:31.917369 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:31.917644 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:35.183856 14149 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:00:35.184077 14149 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:00:35.184783 14149 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:00:35.185914 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:00:35.187213 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:00:35.187395 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:00:35.187515 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:00:35.187764 14149 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:00:36.920933 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:36.921139 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:41.924401 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:41.924592 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:46.927944 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:46.928153 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:51.931097 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:51.931282 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:00:52.277201 14178 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:00:52.277464 14178 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:00:52.278388 14178 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:00:52.279060 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:00:52.280539 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:00:52.280722 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:00:52.280851 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:00:52.281077 14178 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:00:56.934434 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:00:56.934613 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:01.937747 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:01.937935 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:06.941620 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:06.941803 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:11.005796 14209 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:01:11.006026 14209 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:01:11.006912 14209 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:01:11.007929 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:01:11.009879 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:01:11.010054 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:01:11.010174 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:01:11.010409 14209 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:01:11.944592 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:11.944785 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251213 21:01:12.556705 12361 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (179 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:01:12.556845 12501 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (179 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:01:12.556867 12641 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (179 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:01:12.592181 14215 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:01:12.592360 14215 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:01:12.593072 14215 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:01:12.595217 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:01:12.595510 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:01:12.596145 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:01:12.596556 14215 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:01:16.947841 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:16.948031 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:21.951157 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:21.951352 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:26.954352 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:26.954550 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:31.920197 14248 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:01:31.920432 14248 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:01:31.921432 14248 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:01:31.922427 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20251213 21:01:31.924026 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:01:31.924223 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:01:31.924353 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:01:31.924618 14248 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:01:31.927059 13540 maintenance_manager.cc:419] P f2fb6eff9fdc4139b18baa7e29377276: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033360
I20251213 21:01:31.958067 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:31.958313 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:31.977213 13539 maintenance_manager.cc:643] P f2fb6eff9fdc4139b18baa7e29377276: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.049s user 0.019s sys 0.005s Metrics: {"bytes_written":12221,"cfile_init":1,"dirs.queue_time_us":419,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":885,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":628,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":27,"thread_start_us":1016,"threads_started":2,"wal-append.queue_time_us":844}
I20251213 21:01:31.978513 13540 maintenance_manager.cc:419] P f2fb6eff9fdc4139b18baa7e29377276: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1335 bytes on disk
I20251213 21:01:31.979286 13539 maintenance_manager.cc:643] P f2fb6eff9fdc4139b18baa7e29377276: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20251213 21:01:36.961198 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:36.961405 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:39.779110 14264 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:01:39.779336 14264 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:01:39.780071 14264 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:01:39.781177 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:01:39.782645 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:01:39.782871 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:01:39.782989 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:01:39.783300 14264 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:01:41.964313 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:41.964483 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:46.967715 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:46.967897 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:51.971550 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:51.971732 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:53.200459 14307 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:01:53.200690 14307 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:01:53.201471 14307 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:01:53.202559 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:01:53.203733 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:01:53.203918 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:01:53.204046 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:01:53.204339 14307 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:01:56.974808 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:01:56.975000 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:01:59.235069 14343 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:01:59.235293 14343 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:01:59.236055 14343 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:01:59.238124 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:01:59.238124 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:01:59.239498 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:01:59.239694 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:01:59.239809 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:01:59.240128 14343 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:02:01.978009 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:01.978207 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:06.980960 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:06.981138 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:11.984287 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:11.984490 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251213 21:02:12.705031 12361 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (299 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:02:12.705299 12641 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (299 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:02:12.705416 12501 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (299 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:02:14.951409 14428 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:02:14.951640 14428 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:02:14.952472 14428 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:02:14.953449 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:02:14.954746 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:02:14.954895 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:02:14.955027 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:02:14.955287 14428 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:02:16.987306 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:16.987493 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:21.990599 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:21.990777 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:26.993724 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:26.993916 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:28.208681 14502 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:02:28.208886 14502 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:02:28.209636 14502 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:02:28.210593 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:02:28.211750 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:02:28.211910 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:02:28.212042 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:02:28.212337 14502 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:02:31.996802 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:31.996997 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:37.000173 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:37.000366 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:38.392731 14554 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:02:38.392959 14554 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:02:38.393731 14554 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:02:38.394752 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:02:38.395860 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:02:38.396008 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:02:38.396122 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:02:38.396390 14554 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:02:42.003403 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:42.003579 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:47.006959 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:47.007162 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:52.009934 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:52.010118 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:02:56.460794 14718 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:02:56.461012 14718 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:02:56.461911 14718 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:02:56.462949 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:02:56.464571 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:02:56.464736 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:02:56.464881 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:02:56.465127 14718 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:02:57.012493 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:02:57.012661 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:02.015169 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:02.015362 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:07.018362 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:07.018555 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:09.938099 14872 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:03:09.938302 14872 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:03:09.938902 14872 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:03:09.939857 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:03:09.941114 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:03:09.941270 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:03:09.941422 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:03:09.941637 14872 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:03:12.021284 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:12.021499 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251213 21:03:12.854501 12641 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (419 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:03:12.925755 12361 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (419 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:03:12.926132 12503 heartbeater.cc:646] Failed to heartbeat to 127.11.111.61:39203 (433 consecutive failures): Network error: Failed to ping master at 127.11.111.61:39203: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:03:17.024547 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:17.024750 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:22.027923 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:22.028139 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:22.911288 15020 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:03:22.911530 15020 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:03:22.912194 15020 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:03:22.913291 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:03:22.914592 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:03:22.914772 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:03:22.914865 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:03:22.915102 15020 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:03:27.031252 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:27.031450 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:32.034143 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:32.034329 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:34.141274 15153 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:03:34.141506 15153 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:03:34.142223 15153 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:03:34.143175 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:03:34.144461 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:03:34.144601 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:03:34.144769 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:03:34.145057 15153 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:03:37.037436 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:37.037634 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:42.040531 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:42.040721 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:46.133908 15281 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:03:46.134125 15281 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:03:46.134765 15281 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:03:46.135928 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:03:46.137091 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:03:46.137255 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:03:46.137360 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:03:46.137619 15281 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:03:47.043246 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:47.043426 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:52.046342 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:52.046540 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:03:57.049369 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:03:57.049592 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:02.052457 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:02.052636 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:04.674455 15479 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:04:04.674654 15479 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:04:04.675467 15479 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:04:04.676587 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:04:04.677870 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:04:04.678049 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
I20251213 21:04:04.678141 13538 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:04:04.678350 15479 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:04:07.055500 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:07.056211 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:12.059077 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:12.059279 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251213 21:04:12.999884 12641 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (539 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:04:13.198429 12501 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (539 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:04:13.199220 12363 heartbeater.cc:646] Failed to heartbeat to 127.11.111.61:39203 (553 consecutive failures): Network error: Failed to ping master at 127.11.111.61:39203: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:04:13.199224 12361 heartbeater.cc:646] Failed to heartbeat to 127.11.111.60:36039 (539 consecutive failures): Network error: Failed to ping master at 127.11.111.60:36039: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:04:17.062376 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:17.062562 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:22.066007 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:22.066649 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:24.667240 15707 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:04:24.667495 15707 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:04:24.668418 15707 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:04:24.669275 13538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:04:24.670686 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:04:24.670910 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:04:24.671110 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:04:24.671450 15707 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:04:27.069276 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:27.069490 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:32.072098 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:32.072295 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:34.409184 15813 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:04:34.409406 15813 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:04:34.410185 15813 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:04:34.411139 13537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251213 21:04:34.412170 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:04:34.412323 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:04:34.412442 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:04:34.412685 15813 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:04:37.075311 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:37.075512 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251213 21:04:38.792459 15864 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6736d1c95f384b9098c089a20b76e9ae)
I20251213 21:04:38.792691 15864 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb6eff9fdc4139b18baa7e29377276" member_type: VOTER last_known_addr { host: "127.11.111.62" port: 33723 } } peers { permanent_uuid: "68c0fc24cab047dcaa4cae2dbd628626" member_type: VOTER last_known_addr { host: "127.11.111.61" port: 39203 } } peers { permanent_uuid: "6736d1c95f384b9098c089a20b76e9ae" member_type: VOTER last_known_addr { host: "127.11.111.60" port: 36039 } }
I20251213 21:04:38.793511 15864 leader_election.cc:290] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203), 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039)
W20251213 21:04:38.799364 13538 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 68c0fc24cab047dcaa4cae2dbd628626 (127.11.111.61:39203): Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111)
W20251213 21:04:38.799577 13537 leader_election.cc:336] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6736d1c95f384b9098c089a20b76e9ae (127.11.111.60:36039): Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
I20251213 21:04:38.799693 13537 leader_election.cc:304] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fb6eff9fdc4139b18baa7e29377276; no voters: 6736d1c95f384b9098c089a20b76e9ae, 68c0fc24cab047dcaa4cae2dbd628626
I20251213 21:04:38.800047 15864 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f2fb6eff9fdc4139b18baa7e29377276 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251213 21:04:39.516796 13540 maintenance_manager.cc:419] P f2fb6eff9fdc4139b18baa7e29377276: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1335 bytes on disk
I20251213 21:04:39.521739 13539 maintenance_manager.cc:643] P f2fb6eff9fdc4139b18baa7e29377276: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":603,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":110,"thread_start_us":729,"threads_started":2,"wal-append.queue_time_us":328}
I20251213 21:04:42.078485 13519 master_runner.cc:186] Error getting master registration for 127.11.111.61:39203: Network error: Client connection negotiation failed: client connection to 127.11.111.61:39203: connect: Connection refused (error 111),
I20251213 21:04:42.078701 13519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251213 21:04:43.057067 12650 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:04:43.057353 12650 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:04:43.057493 12650 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
W20251213 21:04:43.057621 12650 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
F20251213 21:04:43.068423 12661 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_32850fadebae44e797c016b5161038ab on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1765659883 (unix time) try "date -d @1765659883" if you are using GNU date ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075973 12668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075973 12668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076112 12673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075973 12668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076112 12673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076179 12669 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075973 12668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076112 12673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076179 12669 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076253 12682 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075973 12668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076112 12673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076179 12669 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076253 12682 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076275 12665 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_60f4a42abdd34e9fbfa1e25215d9fba5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075973 12668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076112 12673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076179 12669 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076253 12682 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076275 12665 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_60f4a42abdd34e9fbfa1e25215d9fba5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076310 12656 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9857f31a4d1e42b78febc6e2e68587fc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800002dbc) received by PID 11708 (TID 0x7fc078baf700) from PID 11708; stack trace: ***
F20251213 21:04:43.068483 12663 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9e21ceee05264206866e8b0d62a537c9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.068681 12666 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069795 12677 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069914 12667 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.069973 12679 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070206 12670 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.070252 12678 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071401 12662 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0667e63914504edaa41a24b9c799bfba on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071676 12664 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5009d0ba1a394917a7101259634328ef on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071806 12676 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.071807 12672 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.072952 12660 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_eb562e37a81541048c6a9757feca4265 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073077 12659 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_29cded03ad224e8d809f516d66e903dc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.073451 12671 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.074731 12658 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_cc9c8f278dee46108580b4455e30b213 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075893 12674 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.075973 12668 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076112 12673 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076179 12669 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076253 12682 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076275 12665 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_60f4a42abdd34e9fbfa1e25215d9fba5 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.076310 12656 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9857f31a4d1e42b78febc6e2e68587fc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)F20251213 21:04:43.077613 12657 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e448c113874f43c6b7a38f4b918a08a6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.11.111.62:33723,127.11.111.61:39203,127.11.111.60:36039, num_attempts: 771) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.11.111.60:36039: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fc08df72980 (unknown) at ??:0
@ 0x7fc086d47fb7 gsignal at ??:0
@ 0x7fc086d49921 abort at ??:0
@ 0x7fc089a7fdcd google::LogMessage::Fail() at ??:0
@ 0x7fc089a83b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc089a7f7cc google::LogMessage::Flush() at ??:0
@ 0x7fc089a80f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b582 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
@ 0x44aaf4 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44a9ad std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x44a96a _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x44a91e std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x44a659 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7fc0873cf6df (unknown) at ??:0
@ 0x7fc08df676db start_thread at ??:0
@ 0x7fc086e2a71f clone at ??:0