CatalogManagerConfigurations/MasterStressTest.Test/2: Unrecognized error type. Please see the error log for more information.

Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN      ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250111 08:58:25.598131 16466 test_util.cc:274] Using random seed: 1778806409
2025-01-11T08:58:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-11T08:58:25Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2kHLUS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2kHLUS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:	 jdbc:derby:/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/metadb;create=true
Metastore Connection Driver :	 org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:	 APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 

 
 
 
 

 
 
 
 


 
 
 

 
 
 


 
 
 
 
 
 
 


 
 
 

 
 
 

 
 

 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 


 
 
 
 
 
 



 
 
 
 
 


 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 


Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/security.properties
2025-01-11 08:58:31: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2kHLUS/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task2kHLUS/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250111 08:58:39.099535 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.190:46487
--webserver_interface=127.16.20.190
--webserver_port=0
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 08:58:39.409044 16707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:39.409356 16707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:39.409474 16707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:39.417028 16707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:39.417143 16707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:39.417214 16707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:39.417281 16707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:39.426529 16707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.190:46487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.20.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:39.428068 16707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:39.429741 16707 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:39.440338 16712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:39.440973 16713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:39.441926 16715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:39.442793 16707 server_base.cc:1034] running on GCE node
I20250111 08:58:39.443621 16707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:39.445168 16707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:39.446404 16707 hybrid_clock.cc:648] HybridClock initialized: now 1736585919446324 us; error 55 us; skew 500 ppm
I20250111 08:58:39.446738 16707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:39.456411 16707 webserver.cc:458] Webserver started at http://127.16.20.190:38281/ using document root <none> and password file <none>
I20250111 08:58:39.457202 16707 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:39.457326 16707 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:39.457609 16707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 08:58:39.460208 16707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/instance:
uuid: "c77238c1f3c9423caf1235b18b67ab2e"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:39.460971 16707 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal/instance:
uuid: "c77238c1f3c9423caf1235b18b67ab2e"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:39.466132 16707 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20250111 08:58:39.469553 16721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:39.470943 16707 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20250111 08:58:39.471108 16707 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
uuid: "c77238c1f3c9423caf1235b18b67ab2e"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:39.471315 16707 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:39.513896 16707 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:39.514783 16707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:39.515069 16707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:39.531652 16707 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.190:46487
I20250111 08:58:39.531677 16772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.190:46487 every 8 connection(s)
I20250111 08:58:39.533108 16707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/info.pb
I20250111 08:58:39.536929 16773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:39.542984 16773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } has no permanent_uuid. Determining permanent_uuid...
I20250111 08:58:39.543932 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 16707
I20250111 08:58:39.544240 16466 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal/instance
I20250111 08:58:39.547040 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--webserver_interface=127.16.20.189
--webserver_port=0
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 08:58:39.560006 16773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } has no permanent_uuid. Determining permanent_uuid...
W20250111 08:58:39.561177 16724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:58:39.562685 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:39.612000 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } attempt: 1
W20250111 08:58:39.613875 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:39.682121 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } attempt: 2
W20250111 08:58:39.683943 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:39.775169 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } attempt: 3
W20250111 08:58:39.777034 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:58:39.863725 16776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:39.864054 16776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:39.864182 16776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:39.871820 16776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:39.871953 16776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:39.872030 16776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:39.872109 16776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:39.881414 16776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.20.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:39.883004 16776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:39.884707 16776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:39.895006 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:39.895678 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:39.896234 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:39.897185 16776 server_base.cc:1034] running on GCE node
I20250111 08:58:39.897917 16776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:39.899320 16776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:39.900530 16776 hybrid_clock.cc:648] HybridClock initialized: now 1736585919900465 us; error 38 us; skew 500 ppm
I20250111 08:58:39.900851 16776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:39.903023 16776 webserver.cc:458] Webserver started at http://127.16.20.189:43097/ using document root <none> and password file <none>
I20250111 08:58:39.903749 16776 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:39.903862 16776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:39.904140 16776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 08:58:39.908106 16776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/instance:
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:39.908905 16776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal/instance:
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:39.913962 16776 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20250111 08:58:39.917253 16791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:39.918531 16776 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20250111 08:58:39.918685 16776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:39.918869 16776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:39.920243 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } attempt: 4
W20250111 08:58:39.922135 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:39.939849 16776 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:39.940750 16776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:39.941030 16776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:39.957604 16776 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.189:35063
I20250111 08:58:39.957623 16842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.189:35063 every 8 connection(s)
I20250111 08:58:39.958915 16776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
I20250111 08:58:39.962127 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 16776
I20250111 08:58:39.962282 16466 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal/instance
I20250111 08:58:39.962602 16843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:39.963363 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--webserver_interface=127.16.20.188
--webserver_port=0
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 08:58:39.968421 16843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } has no permanent_uuid. Determining permanent_uuid...
I20250111 08:58:39.981357 16843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } has no permanent_uuid. Determining permanent_uuid...
I20250111 08:58:39.989905 16843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } has no permanent_uuid. Determining permanent_uuid...
W20250111 08:58:39.991087 16793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
W20250111 08:58:39.992590 16843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:40.041903 16843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 1
W20250111 08:58:40.043838 16843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:40.112044 16843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 2
W20250111 08:58:40.114070 16843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:40.205277 16843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 3
W20250111 08:58:40.207114 16843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:40.221362 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } attempt: 5
I20250111 08:58:40.227856 16773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } has no permanent_uuid. Determining permanent_uuid...
W20250111 08:58:40.229682 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:40.280881 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 1
W20250111 08:58:40.282621 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
W20250111 08:58:40.282610 16844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:40.282927 16844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:40.283042 16844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:40.290731 16844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:40.290848 16844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:40.290918 16844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:40.290985 16844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:40.300251 16844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.20.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:40.301827 16844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:40.303468 16844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:40.313812 16852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:40.314447 16855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:40.315361 16853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:40.316161 16844 server_base.cc:1034] running on GCE node
I20250111 08:58:40.316959 16844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:40.318341 16844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:40.319540 16844 hybrid_clock.cc:648] HybridClock initialized: now 1736585920319481 us; error 35 us; skew 500 ppm
I20250111 08:58:40.319854 16844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:40.322033 16844 webserver.cc:458] Webserver started at http://127.16.20.188:41249/ using document root <none> and password file <none>
I20250111 08:58:40.322737 16844 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:40.322850 16844 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:40.323138 16844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 08:58:40.325695 16844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/instance:
uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
format_stamp: "Formatted at 2025-01-11 08:58:40 on dist-test-slave-s924"
I20250111 08:58:40.326396 16844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal/instance:
uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
format_stamp: "Formatted at 2025-01-11 08:58:40 on dist-test-slave-s924"
I20250111 08:58:40.331182 16844 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20250111 08:58:40.334354 16861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:40.335644 16844 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20250111 08:58:40.335798 16844 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
format_stamp: "Formatted at 2025-01-11 08:58:40 on dist-test-slave-s924"
I20250111 08:58:40.335987 16844 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:40.349335 16844 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:40.350155 16844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:40.350338 16843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 4
I20250111 08:58:40.350425 16844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:40.350829 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 2
W20250111 08:58:40.352643 16773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
W20250111 08:58:40.352707 16843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:40.367483 16844 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.188:41785
I20250111 08:58:40.367515 16912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.188:41785 every 8 connection(s)
I20250111 08:58:40.368851 16844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
I20250111 08:58:40.372422 16913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:40.377835 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 16844
I20250111 08:58:40.378051 16466 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal/instance
I20250111 08:58:40.378422 16913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } has no permanent_uuid. Determining permanent_uuid...
I20250111 08:58:40.392597 16913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } has no permanent_uuid. Determining permanent_uuid...
I20250111 08:58:40.399789 16913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } has no permanent_uuid. Determining permanent_uuid...
I20250111 08:58:40.412926 16913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap starting.
I20250111 08:58:40.416378 16913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:40.417433 16913 log.cc:826] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:40.419651 16913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: No bootstrap required, opened a new log
I20250111 08:58:40.425987 16913 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.426288 16913 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:40.426398 16913 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Initialized, Role: FOLLOWER
I20250111 08:58:40.426931 16913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.428162 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.428498 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:40.429260 16913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: configured and running, proceeding with master startup.
W20250111 08:58:40.441123 16935 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 08:58:40.441321 16935 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 08:58:40.437901 16844 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: OK, instance_id { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" instance_seqno: 1736585919527032 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250111 08:58:40.446321 16844 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:40.458945 16773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 3
I20250111 08:58:40.470985 16773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Bootstrap starting.
I20250111 08:58:40.475104 16773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:40.476438 16773 log.cc:826] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:40.482976 16773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: No bootstrap required, opened a new log
I20250111 08:58:40.491878 16773 raft_consensus.cc:357] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.492251 16773 raft_consensus.cc:383] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:40.492419 16773 raft_consensus.cc:738] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77238c1f3c9423caf1235b18b67ab2e, State: Initialized, Role: FOLLOWER
I20250111 08:58:40.493132 16773 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.494098 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.494479 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:40.495450 16773 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:58:40.516399 16921 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:40.516695 16921 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
W20250111 08:58:40.517643 16727 tablet.cc:2367] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 08:58:40.518558 16921 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:58:40.520329 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:58:40.520923 16748 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 0.
I20250111 08:58:40.521679 16865 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 
I20250111 08:58:40.522559 16921 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 08:58:40.522701 16921 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:58:40.522802 16921 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:40.527158 16921 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.528884 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e"
I20250111 08:58:40.529098 16748 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:40.537147 16748 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 1.
I20250111 08:58:40.537632 16865 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 
I20250111 08:58:40.538255 16948 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 08:58:40.543633 16921 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 1 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:58:40.561281 16948 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 1 LEADER]: Becoming Leader. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Running, Role: LEADER
I20250111 08:58:40.562104 16948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.570850 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb4ec5faa3472f9cecb23f9b5d2e0f. Latest consensus state: current_term: 1 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.571159 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:40.572584 16953 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:58:40.574811 16953 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:58:40.589715 16748 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Refusing update from remote peer 4acb4ec5faa3472f9cecb23f9b5d2e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 08:58:40.590715 16921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 08:58:40.602609 16867 tablet.cc:2367] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 08:58:40.619094 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb4ec5faa3472f9cecb23f9b5d2e0f. Latest consensus state: current_term: 1 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.619371 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:40.632143 16954 mvcc.cc:204] Tried to move back new op lower bound from 7113055930723377152 to 7113055930629193728. Current Snapshot: MvccSnapshot[applied={T|T < 7113055930723377152}]
I20250111 08:58:40.633298 16956 mvcc.cc:204] Tried to move back new op lower bound from 7113055930723377152 to 7113055930629193728. Current Snapshot: MvccSnapshot[applied={T|T < 7113055930723377152}]
I20250111 08:58:40.636829 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.637075 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:40.638638 16953 catalog_manager.cc:1348] Generated new cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:40.638729 16953 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:58:40.654475 16843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } attempt: 5
I20250111 08:58:40.627480 16707 master_runner.cc:186] Error getting master registration for 127.16.20.189:35063: OK, instance_id { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" instance_seqno: 1736585919952935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250111 08:58:40.656004 16707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:40.665906 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.666141 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:40.684337 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap starting.
I20250111 08:58:40.688725 16843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:40.689127 16953 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 08:58:40.690640 16953 catalog_manager.cc:1505] Loading token signing keys...
W20250111 08:58:40.694911 16961 catalog_manager.cc:1559] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 08:58:40.695016 16961 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 08:58:40.697371 16843 log.cc:826] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:40.707791 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: No bootstrap required, opened a new log
I20250111 08:58:40.718434 16843 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.718770 16843 raft_consensus.cc:383] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:40.718885 16843 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Initialized, Role: FOLLOWER
I20250111 08:58:40.719553 16843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:40.721666 16953 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Generated new TSK 0
I20250111 08:58:40.722636 16953 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:58:40.723870 16953 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:58:40.725010 16953 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250111 08:58:40.729337 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.730258 16843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:58:40.730634 16816 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:40.734202 16816 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Refusing update from remote peer 4acb4ec5faa3472f9cecb23f9b5d2e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 08:58:40.734735 16818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" is_pre_election: true
I20250111 08:58:40.735219 16818 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 1.
I20250111 08:58:40.735497 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
I20250111 08:58:40.738442 16817 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 1.
I20250111 08:58:40.738794 16921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 08:58:40.739830 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:40.741714 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.20.129:0
--local_ip_for_outbound_sockets=127.16.20.129
--webserver_interface=127.16.20.129
--webserver_port=0
--tserver_master_addrs=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 08:58:40.773638 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb4ec5faa3472f9cecb23f9b5d2e0f. Latest consensus state: current_term: 1 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.773941 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:40.774180 16972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:40.774571 16972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:40.842684 16987 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250111 08:58:41.146138 16970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:41.146413 16970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:41.146540 16970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:41.154007 16970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:41.154227 16970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.20.129
I20250111 08:58:41.163220 16970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.20.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.20.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.20.129
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:41.164788 16970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:41.166457 16970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:41.179189 16993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:41.180079 16992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:41.180699 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:41.181388 16970 server_base.cc:1034] running on GCE node
I20250111 08:58:41.181957 16970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:41.182838 16970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:41.184043 16970 hybrid_clock.cc:648] HybridClock initialized: now 1736585921183974 us; error 48 us; skew 500 ppm
I20250111 08:58:41.184321 16970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:41.186385 16970 webserver.cc:458] Webserver started at http://127.16.20.129:37245/ using document root <none> and password file <none>
I20250111 08:58:41.187064 16970 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:41.187186 16970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:41.187460 16970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 08:58:41.189889 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:41.190152 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:41.190196 16970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data/instance:
uuid: "13afd9f6a62a4efd831797a347bb31b2"
format_stamp: "Formatted at 2025-01-11 08:58:41 on dist-test-slave-s924"
I20250111 08:58:41.191392 16970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/instance:
uuid: "13afd9f6a62a4efd831797a347bb31b2"
format_stamp: "Formatted at 2025-01-11 08:58:41 on dist-test-slave-s924"
I20250111 08:58:41.196064 16970 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20250111 08:58:41.199141 17002 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:41.200613 16970 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20250111 08:58:41.200762 16970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal
uuid: "13afd9f6a62a4efd831797a347bb31b2"
format_stamp: "Formatted at 2025-01-11 08:58:41 on dist-test-slave-s924"
I20250111 08:58:41.200927 16970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:41.222620 17007 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:41.222810 17007 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:41.223981 17007 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:41.224581 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" is_pre_election: true
I20250111 08:58:41.224727 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:41.225498 16723 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:58:41.225848 17007 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:58:41.230083 16970 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:41.230865 16970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:41.231117 16970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:41.231986 16970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 08:58:41.233462 16970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 08:58:41.233532 16970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:41.233616 16970 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 08:58:41.233656 16970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:41.261440 16970 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.129:37699
I20250111 08:58:41.261482 17115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.129:37699 every 8 connection(s)
I20250111 08:58:41.262658 16970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/data/info.pb
I20250111 08:58:41.269886 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 16970
I20250111 08:58:41.270087 16466 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/instance
I20250111 08:58:41.277197 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.20.130:0
--local_ip_for_outbound_sockets=127.16.20.130
--webserver_interface=127.16.20.130
--webserver_port=0
--tserver_master_addrs=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 08:58:41.296720 17116 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:58:41.297030 17116 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:41.298095 17116 heartbeater.cc:510] Master 127.16.20.188:41785 requested a full tablet report, sending...
I20250111 08:58:41.298833 17117 heartbeater.cc:346] Connected to a master server at 127.16.20.190:46487
I20250111 08:58:41.298983 17117 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:41.299494 17117 heartbeater.cc:510] Master 127.16.20.190:46487 requested a full tablet report, sending...
I20250111 08:58:41.300280 16878 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:58:41.300879 17118 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:41.301012 17118 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:41.301256 16738 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:58:41.301432 17118 heartbeater.cc:510] Master 127.16.20.189:35063 requested a full tablet report, sending...
I20250111 08:58:41.302201 16878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.20.129:49583
I20250111 08:58:41.303020 16808 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
W20250111 08:58:41.598505 17126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:41.598747 17126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:41.598891 17126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:41.606398 17126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:41.606597 17126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.20.130
I20250111 08:58:41.615170 17126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.20.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.20.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.20.130
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:41.616585 17126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:41.618122 17126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:41.629575 17131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:41.630211 17134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:41.630712 17132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:41.631789 17126 server_base.cc:1034] running on GCE node
I20250111 08:58:41.632472 17126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:41.633630 17126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:41.634851 17126 hybrid_clock.cc:648] HybridClock initialized: now 1736585921634792 us; error 37 us; skew 500 ppm
I20250111 08:58:41.635125 17126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:41.637120 17126 webserver.cc:458] Webserver started at http://127.16.20.130:41397/ using document root <none> and password file <none>
I20250111 08:58:41.637753 17126 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:41.637863 17126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:41.638139 17126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 08:58:41.640491 17126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data/instance:
uuid: "d650c29a4cae41a490fd9aad87215be7"
format_stamp: "Formatted at 2025-01-11 08:58:41 on dist-test-slave-s924"
I20250111 08:58:41.641204 17126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/instance:
uuid: "d650c29a4cae41a490fd9aad87215be7"
format_stamp: "Formatted at 2025-01-11 08:58:41 on dist-test-slave-s924"
I20250111 08:58:41.645669 17126 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20250111 08:58:41.648619 17140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:41.649734 17126 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20250111 08:58:41.649875 17126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal
uuid: "d650c29a4cae41a490fd9aad87215be7"
format_stamp: "Formatted at 2025-01-11 08:58:41 on dist-test-slave-s924"
I20250111 08:58:41.650022 17126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:41.688853 17126 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:41.689630 17126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:41.689903 17126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:41.690784 17126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 08:58:41.691855 17146 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:41.692046 17146 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:41.692375 17126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 08:58:41.692458 17126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20250111 08:58:41.692571 17126 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 08:58:41.692617 17126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:41.693562 17146 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:41.693909 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:41.693905 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:58:41.694536 16795 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:41.694861 17146 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:58:41.697016 16961 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:41.697103 16961 catalog_manager.cc:1553] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: loading cluster ID for follower catalog manager: success
I20250111 08:58:41.700830 16961 catalog_manager.cc:1575] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: acquiring CA information for follower catalog manager: success
I20250111 08:58:41.703197 16961 catalog_manager.cc:1603] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:58:41.723453 17126 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.130:35105
I20250111 08:58:41.723486 17253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.130:35105 every 8 connection(s)
I20250111 08:58:41.724726 17126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/data/info.pb
I20250111 08:58:41.731817 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 17126
I20250111 08:58:41.732015 16466 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/instance
I20250111 08:58:41.736922 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.20.131:0
--local_ip_for_outbound_sockets=127.16.20.131
--webserver_interface=127.16.20.131
--webserver_port=0
--tserver_master_addrs=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250111 08:58:41.741791 17256 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:41.742115 17256 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:41.743064 17256 heartbeater.cc:510] Master 127.16.20.189:35063 requested a full tablet report, sending...
I20250111 08:58:41.744750 16808 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:41.747574 17254 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:58:41.747702 17254 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:41.748178 17254 heartbeater.cc:510] Master 127.16.20.188:41785 requested a full tablet report, sending...
I20250111 08:58:41.749034 16878 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:41.749943 16878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.20.130:37179
I20250111 08:58:41.749997 17255 heartbeater.cc:346] Connected to a master server at 127.16.20.190:46487
I20250111 08:58:41.750099 17255 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:41.750535 17255 heartbeater.cc:510] Master 127.16.20.190:46487 requested a full tablet report, sending...
I20250111 08:58:41.751605 16738 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:41.804637 17116 heartbeater.cc:502] Master 127.16.20.188:41785 was elected leader, sending a full tablet report...
I20250111 08:58:41.847144 16987 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:41.847287 16987 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: loading cluster ID for follower catalog manager: success
I20250111 08:58:41.851052 16987 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: acquiring CA information for follower catalog manager: success
I20250111 08:58:41.853360 16987 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250111 08:58:42.049631 17261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:42.049898 17261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:42.050031 17261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:42.057535 17261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:42.057750 17261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.20.131
I20250111 08:58:42.066649 17261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.20.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.20.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.20.131
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:42.068079 17261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:42.069677 17261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:42.082403 17267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:42.083292 17268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:42.083942 17270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:42.084666 17261 server_base.cc:1034] running on GCE node
I20250111 08:58:42.085227 17261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:42.086066 17261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:42.087287 17261 hybrid_clock.cc:648] HybridClock initialized: now 1736585922087211 us; error 57 us; skew 500 ppm
I20250111 08:58:42.087620 17261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:42.090142 17261 webserver.cc:458] Webserver started at http://127.16.20.131:33601/ using document root <none> and password file <none>
I20250111 08:58:42.091029 17261 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:42.091187 17261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:42.091533 17261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 08:58:42.094882 17261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data/instance:
uuid: "25e23fc83774472986af5c3f81f591fe"
format_stamp: "Formatted at 2025-01-11 08:58:42 on dist-test-slave-s924"
I20250111 08:58:42.095788 17261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/instance:
uuid: "25e23fc83774472986af5c3f81f591fe"
format_stamp: "Formatted at 2025-01-11 08:58:42 on dist-test-slave-s924"
I20250111 08:58:42.102048 17261 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20250111 08:58:42.106104 17276 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:42.107573 17261 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20250111 08:58:42.107736 17261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal
uuid: "25e23fc83774472986af5c3f81f591fe"
format_stamp: "Formatted at 2025-01-11 08:58:42 on dist-test-slave-s924"
I20250111 08:58:42.107940 17261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:42.142920 17261 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:42.143942 17261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:42.144282 17261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:42.145422 17261 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 08:58:42.147382 17261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 08:58:42.147475 17261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:42.147595 17261 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 08:58:42.147656 17261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:42.182790 17261 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.131:33627
I20250111 08:58:42.182809 17388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.131:33627 every 8 connection(s)
I20250111 08:58:42.184545 17261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/data/info.pb
I20250111 08:58:42.188766 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 17261
I20250111 08:58:42.188921 16466 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/instance
I20250111 08:58:42.213507 17391 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:42.213831 17391 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:42.214815 17391 heartbeater.cc:510] Master 127.16.20.189:35063 requested a full tablet report, sending...
I20250111 08:58:42.215170 17390 heartbeater.cc:346] Connected to a master server at 127.16.20.190:46487
I20250111 08:58:42.215297 17390 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:42.215756 17390 heartbeater.cc:510] Master 127.16.20.190:46487 requested a full tablet report, sending...
I20250111 08:58:42.216526 16808 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:42.216732 16738 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:42.217736 17389 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:58:42.217842 17389 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:42.218207 17389 heartbeater.cc:510] Master 127.16.20.188:41785 requested a full tablet report, sending...
I20250111 08:58:42.218923 16878 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:42.219647 16878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.20.131:49239
I20250111 08:58:42.228689 17396 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:42.228857 17396 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:42.229923 17396 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:42.230001 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:58:42.230165 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:42.230700 16793 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:42.230852 16466 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250111 08:58:42.230957 17396 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:58:42.252076 17254 heartbeater.cc:502] Master 127.16.20.188:41785 was elected leader, sending a full tablet report...
I20250111 08:58:42.258678 17416 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:42.258864 17416 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:42.259583 17416 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:42.259393 16878 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.259665 16875 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_971c3c3dd7714c618c4d251288537810"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.260998 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" is_pre_election: true
I20250111 08:58:42.261231 16817 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77238c1f3c9423caf1235b18b67ab2e in term 1.
I20250111 08:58:42.261935 16877 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.262365 16876 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.262902 16724 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 
I20250111 08:58:42.263190 17416 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 08:58:42.263291 17416 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:42.263374 17416 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 1 FOLLOWER]: Advancing to term 2
I20250111 08:58:42.266094 17416 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:42.267831 16874 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_7d1113f20be344b0802727c6883a0652"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.268623 16872 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.268676 16871 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20250111 08:58:42.268987 16878 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bd323ab641234f2ebfe5a90526f93cdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 08:58:42.269346 16869 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.269430 16870 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.269891 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:42.270342 16873 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:42.271054 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
I20250111 08:58:42.271405 16817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
I20250111 08:58:42.271616 16817 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 1 FOLLOWER]: Advancing to term 2
W20250111 08:58:42.271755 16875 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_971c3c3dd7714c618c4d251288537810 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 08:58:42.272708 17416 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 2 election: Requested vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:42.274264 16817 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77238c1f3c9423caf1235b18b67ab2e in term 2.
I20250111 08:58:42.275111 16724 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f
I20250111 08:58:42.275516 17416 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 FOLLOWER]: Leader election won for term 2
I20250111 08:58:42.286360 17416 raft_consensus.cc:695] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 LEADER]: Becoming Leader. State: Replica: c77238c1f3c9423caf1235b18b67ab2e, State: Running, Role: LEADER
I20250111 08:58:42.287014 17416 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:42.292716 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77238c1f3c9423caf1235b18b67ab2e. Latest consensus state: current_term: 2 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:42.293038 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:42.293727 17433 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:58:42.296154 16466 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task2kHLUS/build/asan/bin/kudu pid 16776
I20250111 08:58:42.296420 17433 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:58:42.297485 17433 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:42.297623 17433 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:58:42.297971 16838 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 08:58:42.299408 17433 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:58:42.300442 17433 catalog_manager.cc:5910] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Loaded TSK: 0
I20250111 08:58:42.301548 17433 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:58:42.302593 17433 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:58:42.303997 17433 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250111 08:58:42.312609 16747 raft_consensus.cc:1235] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 LEADER]: Rejecting Update request from peer 4acb4ec5faa3472f9cecb23f9b5d2e0f for earlier term 1. Current term is 2. Ops: []
I20250111 08:58:42.313185 17441 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 08:58:42.313725 17441 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Peer c77238c1f3c9423caf1235b18b67ab2e log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250111 08:58:42.313944 17442 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 1 LEADER]: Stepping down as leader of term 1
I20250111 08:58:42.314052 17442 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Running, Role: LEADER
I20250111 08:58:42.314429 17442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:42.314971 17442 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 1 FOLLOWER]: Advancing to term 2
I20250111 08:58:42.436038 16466 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2kHLUS/build/asan/bin/kudu with pid 16776
I20250111 08:58:42.444408 16935 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:58:42.664841 16887 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Refusing update from remote peer c77238c1f3c9423caf1235b18b67ab2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250111 08:58:42.665818 17416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 08:58:42.666020 16878 catalog_manager.cc:2149] created HMS catalog entry for table  [id=5a8fbc5594ed46b2acbdd24dccfb3b39]
W20250111 08:58:42.667394 16878 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:42.680895 17442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77238c1f3c9423caf1235b18b67ab2e. Latest consensus state: current_term: 2 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:42.681183 17442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
W20250111 08:58:42.681341 16724 consensus_peers.cc:487] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e -> Peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Couldn't send request to peer 77aa4f8aaf704f1898426e17c6fbe28f. Status: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 08:58:42.683142 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:42.683372 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:42.684180 17442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:42.684384 17442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:42.702013 16875 catalog_manager.cc:2149] created HMS catalog entry for table  [id=537f7f2646a04bd4bd13025e8f4373d5]
W20250111 08:58:42.702752 16875 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
W20250111 08:58:42.704054 16876 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_518dba2592a54eddbd2b2caec0bb4cfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.704218 16877 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a0ad18d08e114a01ac92b944b0d9efec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.704377 16874 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7d1113f20be344b0802727c6883a0652 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.704464 16871 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.704663 16872 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5f3d49b148aa4b76b52aff24d64f1ad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.705497 16873 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_047f7156c794468f8b656960fc0edb9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.706182 16870 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_da4abbdb96ec46e1a5a37a5845eeb580 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.706866 16869 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_413c709bdd944e2d98cb0e89f0a029ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:42.719556 17391 heartbeater.cc:643] Failed to heartbeat to 127.16.20.189:35063 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:42.721410 17389 heartbeater.cc:502] Master 127.16.20.188:41785 was elected leader, sending a full tablet report...
W20250111 08:58:42.764601 17256 heartbeater.cc:643] Failed to heartbeat to 127.16.20.189:35063 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:42.806649 17117 heartbeater.cc:502] Master 127.16.20.190:46487 was elected leader, sending a full tablet report...
W20250111 08:58:42.949661 17118 heartbeater.cc:643] Failed to heartbeat to 127.16.20.189:35063 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:43.078394 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--webserver_interface=127.16.20.189
--webserver_port=43097
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 08:58:43.183526 16878 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=5a8fbc5594ed46b2acbdd24dccfb3b39]
I20250111 08:58:43.184126 17442 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:58:43.184311 17442 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
W20250111 08:58:43.184989 16878 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.16.20.129:49583 (request call id 3) took 878 ms (client timeout 1000 ms). Trace:
W20250111 08:58:43.185138 16878 rpcz_store.cc:269] 0111 08:58:42.306878 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:43.184405 (+877527us) service_pool.cc:225] Handling call
0111 08:58:43.184928 (+   523us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122752}
I20250111 08:58:43.186122 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:58:43.185111 17442 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:43.187929 17398 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.188:41785)
I20250111 08:58:43.198279 16738 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:43.205420 16959 catalog_manager.cc:2710] Deleting table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5a8fbc5594ed46b2acbdd24dccfb3b39] in response to Hive Metastore notification log event 3
W20250111 08:58:43.205946 16959 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 3 DROP_TABLE default.table_bd323ab641234f2ebfe5a90526f93cdd: Not found: the table does not exist: table_id: "5a8fbc5594ed46b2acbdd24dccfb3b39" table_name: "default.table_bd323ab641234f2ebfe5a90526f93cdd"
W20250111 08:58:43.215309 16864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:58:43.217447 16864 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:43.217713 16864 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:43.219636 17442 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 08:58:43.219692 17390 heartbeater.cc:502] Master 127.16.20.190:46487 was elected leader, sending a full tablet report...
I20250111 08:58:43.255580 17255 heartbeater.cc:502] Master 127.16.20.190:46487 was elected leader, sending a full tablet report...
W20250111 08:58:43.269505 16738 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bd323ab641234f2ebfe5a90526f93cdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 08:58:43.289566 16875 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=537f7f2646a04bd4bd13025e8f4373d5]
I20250111 08:58:43.289944 16875 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 4) took 1031 ms. Trace:
I20250111 08:58:43.290067 16875 rpcz_store.cc:276] 0111 08:58:42.258862 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.259416 (+   554us) service_pool.cc:225] Handling call
0111 08:58:42.271847 (+ 12431us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.272266 (+   419us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:42.701959 (+429693us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:42.702637 (+   678us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:42.702641 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:42.702855 (+   214us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:43.289898 (+587043us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:42.702576 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:42.702585 (+     9us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:42.702590 (+     5us) write_op.cc:454] Released schema lock
0111 08:58:42.702592 (+     2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":873726,"Hive Metastore.run_cpu_time_us":493,"Hive Metastore.run_wall_time_us":142096,"child_traces":[["op",{}]]}
I20250111 08:58:43.295815 16737 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_971c3c3dd7714c618c4d251288537810"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:43.339766 16874 catalog_manager.cc:2149] created HMS catalog entry for table  [id=3f0673839c43477080a94acfef4b5125]
W20250111 08:58:43.340860 16874 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:43.379657 16738 catalog_manager.cc:2149] created HMS catalog entry for table  [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:58:43.393417 16738 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34578:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:58:43.400625 16959 catalog_manager.cc:2710] Deleting table default.table_971c3c3dd7714c618c4d251288537810 [id=537f7f2646a04bd4bd13025e8f4373d5] in response to Hive Metastore notification log event 4
W20250111 08:58:43.400914 16959 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 4 DROP_TABLE default.table_971c3c3dd7714c618c4d251288537810: Not found: the table does not exist: table_id: "537f7f2646a04bd4bd13025e8f4373d5" table_name: "default.table_971c3c3dd7714c618c4d251288537810"
I20250111 08:58:43.401742 16876 catalog_manager.cc:2149] created HMS catalog entry for table  [id=eb5afce7057a493aae25a704cda87ea4]
W20250111 08:58:43.402514 16876 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
W20250111 08:58:43.427626 16737 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_971c3c3dd7714c618c4d251288537810 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 08:58:43.468196 17324 tablet_service.cc:1467] Processing CreateTablet for tablet c5907ff7733044929dbfccb9a9727df1 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.469745 17324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5907ff7733044929dbfccb9a9727df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.475203 17189 tablet_service.cc:1467] Processing CreateTablet for tablet c5907ff7733044929dbfccb9a9727df1 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.476856 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5907ff7733044929dbfccb9a9727df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.483865 16871 catalog_manager.cc:2149] created HMS catalog entry for table  [id=f00998fbd79b4a5baec97891252d6a98]
W20250111 08:58:43.484865 16871 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:43.488423 17051 tablet_service.cc:1467] Processing CreateTablet for tablet c5907ff7733044929dbfccb9a9727df1 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.489370 17471 tablet_bootstrap.cc:492] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:43.489905 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5907ff7733044929dbfccb9a9727df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.492769 17471 tablet_bootstrap.cc:654] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.494048 17471 log.cc:826] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:43.503371 17471 tablet_bootstrap.cc:492] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:43.503747 17471 ts_tablet_manager.cc:1397] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.007s
I20250111 08:58:43.512614 17471 raft_consensus.cc:357] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.513402 17471 raft_consensus.cc:383] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.513567 17471 raft_consensus.cc:738] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.514367 17471 consensus_queue.cc:260] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.532824 17475 tablet_bootstrap.cc:492] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:43.536471 17475 tablet_bootstrap.cc:654] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.540105 17475 log.cc:826] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:43.558918 17475 tablet_bootstrap.cc:492] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:43.559016 16736 master_service.cc:946] ReplaceTablet: received request to replace tablet c5907ff7733044929dbfccb9a9727df1 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.561025 17474 tablet_bootstrap.cc:492] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:43.562017 16735 master_service.cc:946] ReplaceTablet: received request to replace tablet c5907ff7733044929dbfccb9a9727df1 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.562103 16734 master_service.cc:946] ReplaceTablet: received request to replace tablet c5907ff7733044929dbfccb9a9727df1 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.562388 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet c5907ff7733044929dbfccb9a9727df1 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.563236 16732 master_service.cc:946] ReplaceTablet: received request to replace tablet c5907ff7733044929dbfccb9a9727df1 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.567737 17475 ts_tablet_manager.cc:1397] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.035s	user 0.003s	sys 0.008s
I20250111 08:58:43.571204 16736 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet c5907ff7733044929dbfccb9a9727df1
I20250111 08:58:43.575181 16736 catalog_manager.cc:6596] ReplaceTablet: tablet c5907ff7733044929dbfccb9a9727df1 deleted and replaced by tablet cca218fd9cb84cb796bdea731785f440
I20250111 08:58:43.575951 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet c5907ff7733044929dbfccb9a9727df1 with delete_type TABLET_DATA_DELETED (replaced by tablet cca218fd9cb84cb796bdea731785f440) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:43.576565 17475 raft_consensus.cc:357] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.564350 17474 tablet_bootstrap.cc:654] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
W20250111 08:58:43.577708 16724 catalog_manager.cc:4852] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): delete failed for tablet c5907ff7733044929dbfccb9a9727df1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5907ff7733044929dbfccb9a9727df1 already in progress: creating tablet
I20250111 08:58:43.578446 17324 tablet_service.cc:1514] Processing DeleteTablet for tablet c5907ff7733044929dbfccb9a9727df1 with delete_type TABLET_DATA_DELETED (replaced by tablet cca218fd9cb84cb796bdea731785f440) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:43.578850 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet c5907ff7733044929dbfccb9a9727df1 with delete_type TABLET_DATA_DELETED (replaced by tablet cca218fd9cb84cb796bdea731785f440) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:43.580842 17474 log.cc:826] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:43.585121 17188 tablet_service.cc:1467] Processing CreateTablet for tablet cca218fd9cb84cb796bdea731785f440 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.585431 17323 tablet_service.cc:1467] Processing CreateTablet for tablet cca218fd9cb84cb796bdea731785f440 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.585620 17188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca218fd9cb84cb796bdea731785f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.585912 17323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca218fd9cb84cb796bdea731785f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.586397 16737 catalog_manager.cc:2149] created HMS catalog entry for table  [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:58:43.587512 17471 ts_tablet_manager.cc:1428] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.084s	user 0.027s	sys 0.023s
I20250111 08:58:43.590224 16732 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_4b95455ed62a4f51a8aa85b327bc4699"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:43.597724 16737 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34578:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:58:43.598496 17471 tablet_bootstrap.cc:492] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:43.599821 17474 tablet_bootstrap.cc:492] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:43.600157 17471 tablet_bootstrap.cc:654] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.601267 17474 ts_tablet_manager.cc:1397] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.040s	user 0.011s	sys 0.000s
I20250111 08:58:43.602015 17188 tablet_service.cc:1467] Processing CreateTablet for tablet fd69832a13fc426e843ddffa5fe8facb (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.602422 17188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd69832a13fc426e843ddffa5fe8facb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.602885 17323 tablet_service.cc:1467] Processing CreateTablet for tablet fd69832a13fc426e843ddffa5fe8facb (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.603329 17323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd69832a13fc426e843ddffa5fe8facb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.610761 17481 tablet_replica.cc:331] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:43.611828 17481 raft_consensus.cc:2238] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
W20250111 08:58:43.611842 16725 catalog_manager.cc:4852] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): delete failed for tablet c5907ff7733044929dbfccb9a9727df1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5907ff7733044929dbfccb9a9727df1 already in progress: creating tablet
I20250111 08:58:43.584728 17050 tablet_service.cc:1467] Processing CreateTablet for tablet cca218fd9cb84cb796bdea731785f440 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.612712 17471 tablet_bootstrap.cc:492] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:43.612864 17050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca218fd9cb84cb796bdea731785f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.612908 17471 ts_tablet_manager.cc:1397] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.015s	user 0.001s	sys 0.003s
I20250111 08:58:43.613755 17471 raft_consensus.cc:357] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.613935 17471 raft_consensus.cc:383] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.614069 17471 raft_consensus.cc:738] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.614351 17471 consensus_queue.cc:260] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.614840 17471 ts_tablet_manager.cc:1428] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20250111 08:58:43.601431 17051 tablet_service.cc:1467] Processing CreateTablet for tablet fd69832a13fc426e843ddffa5fe8facb (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.616725 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd69832a13fc426e843ddffa5fe8facb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.619828 17471 tablet_bootstrap.cc:492] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
W20250111 08:58:43.620834 17453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:43.621196 17453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:43.621327 17453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250111 08:58:43.621464 17471 tablet_bootstrap.cc:654] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.625180 17481 raft_consensus.cc:2267] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.626756 17481 ts_tablet_manager.cc:1905] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.627243 16872 catalog_manager.cc:2149] created HMS catalog entry for table  [id=30e8d8c675d1451d9f2772e635af7bd4]
W20250111 08:58:43.628062 16872 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:43.628048 17475 raft_consensus.cc:383] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.628170 17475 raft_consensus.cc:738] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.629384 17475 consensus_queue.cc:260] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.634531 17474 raft_consensus.cc:357] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
W20250111 08:58:43.635100 16732 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4b95455ed62a4f51a8aa85b327bc4699 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250111 08:58:43.635674 17453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:43.635792 17453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:43.635859 17453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:43.635937 17453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:43.638128 17475 ts_tablet_manager.cc:1428] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.070s	user 0.012s	sys 0.000s
I20250111 08:58:43.642565 17471 tablet_bootstrap.cc:492] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:43.642765 17471 ts_tablet_manager.cc:1397] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.000s
I20250111 08:58:43.643456 17471 raft_consensus.cc:357] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.643656 17471 raft_consensus.cc:383] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.643723 17471 raft_consensus.cc:738] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.644002 17471 consensus_queue.cc:260] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.645076 17050 tablet_service.cc:1514] Processing DeleteTablet for tablet c5907ff7733044929dbfccb9a9727df1 with delete_type TABLET_DATA_DELETED (replaced by tablet cca218fd9cb84cb796bdea731785f440) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:43.646354 17475 tablet_bootstrap.cc:492] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:43.648029 17475 tablet_bootstrap.cc:654] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.648972 17480 tablet_replica.cc:331] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:43.649185 17471 ts_tablet_manager.cc:1428] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20250111 08:58:43.649430 17480 raft_consensus.cc:2238] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:43.649822 17480 raft_consensus.cc:2267] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.649962 17474 raft_consensus.cc:383] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.651468 17480 ts_tablet_manager.cc:1905] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.640059 17481 ts_tablet_manager.cc:1918] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.653566 17481 log.cc:1198] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/c5907ff7733044929dbfccb9a9727df1
I20250111 08:58:43.654063 17481 ts_tablet_manager.cc:1939] T c5907ff7733044929dbfccb9a9727df1 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:43.659024 17480 ts_tablet_manager.cc:1918] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.659245 17480 log.cc:1198] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/c5907ff7733044929dbfccb9a9727df1
I20250111 08:58:43.659642 17480 ts_tablet_manager.cc:1939] T c5907ff7733044929dbfccb9a9727df1 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:43.660265 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet c5907ff7733044929dbfccb9a9727df1 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:43.650082 17474 raft_consensus.cc:738] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.661201 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet c5907ff7733044929dbfccb9a9727df1 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:43.661789 17474 consensus_queue.cc:260] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.666249 17475 tablet_bootstrap.cc:492] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:43.666428 17475 ts_tablet_manager.cc:1397] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.020s	user 0.000s	sys 0.004s
I20250111 08:58:43.667194 17475 raft_consensus.cc:357] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.667438 17475 raft_consensus.cc:383] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.667550 17475 raft_consensus.cc:738] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.667829 17475 consensus_queue.cc:260] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.668737 17453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.20.189
--webserver_port=43097
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:43.670754 17453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:43.672936 17453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 08:58:43.673770 17475 ts_tablet_manager.cc:1428] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.001s
I20250111 08:58:43.678450 17475 tablet_bootstrap.cc:492] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:43.680397 17475 tablet_bootstrap.cc:654] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.686283 17474 ts_tablet_manager.cc:1428] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.085s	user 0.016s	sys 0.016s
W20250111 08:58:43.689545 17392 tablet.cc:2367] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 08:58:43.691679 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet c5907ff7733044929dbfccb9a9727df1 with delete_type TABLET_DATA_DELETED (replaced by tablet cca218fd9cb84cb796bdea731785f440) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:43.692009 17482 tablet_replica.cc:331] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:43.692440 17482 raft_consensus.cc:2238] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:43.692826 17482 raft_consensus.cc:2267] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.693779 17475 tablet_bootstrap.cc:492] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:43.693949 17475 ts_tablet_manager.cc:1397] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.003s
I20250111 08:58:43.694322 17482 ts_tablet_manager.cc:1905] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.694864 17475 raft_consensus.cc:357] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.695091 17475 raft_consensus.cc:383] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.695171 17475 raft_consensus.cc:738] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.695436 17475 consensus_queue.cc:260] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.695953 17475 ts_tablet_manager.cc:1428] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
W20250111 08:58:43.703002 17492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:43.703809 17491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:43.711256 17482 ts_tablet_manager.cc:1918] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.711421 17482 log.cc:1198] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/c5907ff7733044929dbfccb9a9727df1
I20250111 08:58:43.713598 17474 tablet_bootstrap.cc:492] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:43.716151 17474 tablet_bootstrap.cc:654] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.720703 17482 ts_tablet_manager.cc:1939] T c5907ff7733044929dbfccb9a9727df1 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:43.722467 16873 catalog_manager.cc:2149] created HMS catalog entry for table  [id=39ed1d0a4a7d415b9dba140fb6bbcec2]
W20250111 08:58:43.723376 16873 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:43.725363 17474 tablet_bootstrap.cc:492] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:43.725584 17474 ts_tablet_manager.cc:1397] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.001s
I20250111 08:58:43.726438 17474 raft_consensus.cc:357] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.727595 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet c5907ff7733044929dbfccb9a9727df1 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:43.727982 17474 raft_consensus.cc:383] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.728334 17474 raft_consensus.cc:738] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.728677 17474 consensus_queue.cc:260] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.729305 17474 ts_tablet_manager.cc:1428] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20250111 08:58:43.729620 17474 tablet_bootstrap.cc:492] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:43.736343 17474 tablet_bootstrap.cc:654] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
W20250111 08:58:43.739261 17495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:43.743230 17474 tablet_bootstrap.cc:492] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:43.743608 17453 server_base.cc:1034] running on GCE node
I20250111 08:58:43.743788 17474 ts_tablet_manager.cc:1397] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.008s
I20250111 08:58:43.744732 17453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:43.744809 17474 raft_consensus.cc:357] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.745160 17474 raft_consensus.cc:383] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.745286 17474 raft_consensus.cc:738] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.745602 17474 consensus_queue.cc:260] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.747045 17474 ts_tablet_manager.cc:1428] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20250111 08:58:43.748878 17453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:43.761351 17486 raft_consensus.cc:491] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:43.761569 17486 raft_consensus.cc:513] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.762729 17453 hybrid_clock.cc:648] HybridClock initialized: now 1736585923757657 us; error 5042 us; skew 500 ppm
I20250111 08:58:43.763144 17453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:43.763317 17486 leader_election.cc:290] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
W20250111 08:58:43.766297 17119 tablet.cc:2367] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 08:58:43.771291 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69832a13fc426e843ddffa5fe8facb" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7" is_pre_election: true
I20250111 08:58:43.771732 17209 raft_consensus.cc:2463] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 0.
I20250111 08:58:43.772497 17006 leader_election.cc:304] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7; no voters: 
I20250111 08:58:43.774886 17486 raft_consensus.cc:2798] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 08:58:43.775038 17486 raft_consensus.cc:491] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:58:43.775125 17486 raft_consensus.cc:3054] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:43.778641 17486 raft_consensus.cc:513] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.780200 17486 leader_election.cc:290] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 election: Requested vote from peers d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:43.780465 17453 webserver.cc:458] Webserver started at http://127.16.20.189:43097/ using document root <none> and password file <none>
I20250111 08:58:43.781195 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69832a13fc426e843ddffa5fe8facb" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe" is_pre_election: true
I20250111 08:58:43.781709 17344 raft_consensus.cc:2463] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 0.
I20250111 08:58:43.782218 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69832a13fc426e843ddffa5fe8facb" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe"
I20250111 08:58:43.782445 17343 raft_consensus.cc:3054] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:43.783622 16736 master_service.cc:946] ReplaceTablet: received request to replace tablet fd69832a13fc426e843ddffa5fe8facb from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.784619 17453 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:43.784777 17453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:43.788040 16734 master_service.cc:946] ReplaceTablet: received request to replace tablet cca218fd9cb84cb796bdea731785f440 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.789237 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd69832a13fc426e843ddffa5fe8facb" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7"
I20250111 08:58:43.789426 17209 raft_consensus.cc:3054] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:43.790648 17343 raft_consensus.cc:2463] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 1.
I20250111 08:58:43.791144 17004 leader_election.cc:304] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, 25e23fc83774472986af5c3f81f591fe; no voters: 
I20250111 08:58:43.791422 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet fd69832a13fc426e843ddffa5fe8facb from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.791525 17486 raft_consensus.cc:2798] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 08:58:43.791682 16731 master_service.cc:946] ReplaceTablet: received request to replace tablet cca218fd9cb84cb796bdea731785f440 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.792141 17209 raft_consensus.cc:2463] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 1.
I20250111 08:58:43.795058 16736 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet fd69832a13fc426e843ddffa5fe8facb
I20250111 08:58:43.795516 16736 catalog_manager.cc:6596] ReplaceTablet: tablet fd69832a13fc426e843ddffa5fe8facb deleted and replaced by tablet acfd28cf5c804d88b7877d57a1cc2248
I20250111 08:58:43.796466 16870 catalog_manager.cc:2149] created HMS catalog entry for table  [id=20cc955c3d054cc9b4199caad238843b]
W20250111 08:58:43.797220 16870 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:43.798887 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet fd69832a13fc426e843ddffa5fe8facb with delete_type TABLET_DATA_DELETED (replaced by tablet acfd28cf5c804d88b7877d57a1cc2248) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:43.799451 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet fd69832a13fc426e843ddffa5fe8facb with delete_type TABLET_DATA_DELETED (replaced by tablet acfd28cf5c804d88b7877d57a1cc2248) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:43.799634 17050 tablet_service.cc:1514] Processing DeleteTablet for tablet fd69832a13fc426e843ddffa5fe8facb with delete_type TABLET_DATA_DELETED (replaced by tablet acfd28cf5c804d88b7877d57a1cc2248) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:43.799731 17482 tablet_replica.cc:331] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:43.799887 17480 tablet_replica.cc:331] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:43.799916 17482 raft_consensus.cc:2238] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:43.800088 17482 raft_consensus.cc:2267] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.800782 17482 ts_tablet_manager.cc:1905] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.802217 16736 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_8f0c7f94440643c792a9eed3ea2857f0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:43.802549 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet cca218fd9cb84cb796bdea731785f440 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:43.802845 17481 tablet_replica.cc:331] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:43.803035 17481 raft_consensus.cc:2238] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:43.803179 17481 raft_consensus.cc:2267] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.804077 17481 ts_tablet_manager.cc:1905] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.804998 16734 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet cca218fd9cb84cb796bdea731785f440
I20250111 08:58:43.805447 16734 catalog_manager.cc:6596] ReplaceTablet: tablet cca218fd9cb84cb796bdea731785f440 deleted and replaced by tablet d58685698a294cbe8834cdfcb4d14548
I20250111 08:58:43.806697 17482 ts_tablet_manager.cc:1918] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.806828 17482 log.cc:1198] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/fd69832a13fc426e843ddffa5fe8facb
I20250111 08:58:43.807145 17482 ts_tablet_manager.cc:1939] T fd69832a13fc426e843ddffa5fe8facb P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:43.807651 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet cca218fd9cb84cb796bdea731785f440 with delete_type TABLET_DATA_DELETED (replaced by tablet d58685698a294cbe8834cdfcb4d14548) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:43.808764 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet cca218fd9cb84cb796bdea731785f440 with delete_type TABLET_DATA_DELETED (replaced by tablet d58685698a294cbe8834cdfcb4d14548) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:43.809007 17482 tablet_replica.cc:331] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:43.809017 17050 tablet_service.cc:1514] Processing DeleteTablet for tablet cca218fd9cb84cb796bdea731785f440 with delete_type TABLET_DATA_DELETED (replaced by tablet d58685698a294cbe8834cdfcb4d14548) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:43.809150 17482 raft_consensus.cc:2238] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:43.809322 17482 raft_consensus.cc:2267] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.809975 17482 ts_tablet_manager.cc:1905] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.810295 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet fd69832a13fc426e843ddffa5fe8facb (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:43.815508 17481 ts_tablet_manager.cc:1918] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.815712 17481 log.cc:1198] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/fd69832a13fc426e843ddffa5fe8facb
I20250111 08:58:43.816246 17481 ts_tablet_manager.cc:1939] T fd69832a13fc426e843ddffa5fe8facb P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:43.816919 17189 tablet_service.cc:1467] Processing CreateTablet for tablet acfd28cf5c804d88b7877d57a1cc2248 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.817252 17481 tablet_replica.cc:331] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:43.817299 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acfd28cf5c804d88b7877d57a1cc2248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.817483 17481 raft_consensus.cc:2238] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:43.817718 17481 raft_consensus.cc:2267] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.818202 17323 tablet_service.cc:1467] Processing CreateTablet for tablet acfd28cf5c804d88b7877d57a1cc2248 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.818598 17323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acfd28cf5c804d88b7877d57a1cc2248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.819074 17481 ts_tablet_manager.cc:1905] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.823531 17050 tablet_service.cc:1467] Processing CreateTablet for tablet acfd28cf5c804d88b7877d57a1cc2248 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.823865 17474 tablet_bootstrap.cc:492] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:43.823951 17050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acfd28cf5c804d88b7877d57a1cc2248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.825445 17474 tablet_bootstrap.cc:654] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.828454 17486 raft_consensus.cc:695] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 1 LEADER]: Becoming Leader. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Running, Role: LEADER
I20250111 08:58:43.828888 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet fd69832a13fc426e843ddffa5fe8facb (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:43.828953 17486 consensus_queue.cc:237] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.837836 17453 fs_manager.cc:714] Time spent opening directory manager: real 0.041s	user 0.004s	sys 0.000s
I20250111 08:58:43.838438 17189 tablet_service.cc:1467] Processing CreateTablet for tablet d58685698a294cbe8834cdfcb4d14548 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.838896 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58685698a294cbe8834cdfcb4d14548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.838826 17051 tablet_service.cc:1467] Processing CreateTablet for tablet d58685698a294cbe8834cdfcb4d14548 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.839198 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58685698a294cbe8834cdfcb4d14548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.840934 17482 ts_tablet_manager.cc:1918] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.841068 17482 log.cc:1198] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/cca218fd9cb84cb796bdea731785f440
I20250111 08:58:43.841560 17481 ts_tablet_manager.cc:1918] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.841708 17481 log.cc:1198] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/cca218fd9cb84cb796bdea731785f440
I20250111 08:58:43.842065 17481 ts_tablet_manager.cc:1939] T cca218fd9cb84cb796bdea731785f440 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:43.844013 17482 ts_tablet_manager.cc:1939] T cca218fd9cb84cb796bdea731785f440 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:43.845448 17480 raft_consensus.cc:2238] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 1 LEADER]: Raft consensus shutting down.
I20250111 08:58:43.846099 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet cca218fd9cb84cb796bdea731785f440 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:43.846241 17480 pending_rounds.cc:62] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Trying to abort 1 pending ops.
I20250111 08:58:43.846412 17480 pending_rounds.cc:69] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113055944008241152 op_type: NO_OP noop_request { }
I20250111 08:58:43.846565 17480 raft_consensus.cc:2883] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 08:58:43.846684 17480 raft_consensus.cc:2267] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.846947 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet cca218fd9cb84cb796bdea731785f440 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:43.847651 17475 tablet_bootstrap.cc:492] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:43.847685 17324 tablet_service.cc:1467] Processing CreateTablet for tablet d58685698a294cbe8834cdfcb4d14548 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:43.848069 17324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58685698a294cbe8834cdfcb4d14548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:43.849264 17475 tablet_bootstrap.cc:654] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.853542 17471 tablet_bootstrap.cc:492] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:43.855192 17471 tablet_bootstrap.cc:654] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.859398 16869 catalog_manager.cc:2149] created HMS catalog entry for table  [id=b90143630fb14428b06c0f2828bd6d6a]
I20250111 08:58:43.860143 17480 ts_tablet_manager.cc:1905] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250111 08:58:43.860235 16869 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:43.866894 17480 ts_tablet_manager.cc:1918] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 08:58:43.867085 17480 log.cc:1198] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/fd69832a13fc426e843ddffa5fe8facb
I20250111 08:58:43.867514 17480 ts_tablet_manager.cc:1939] T fd69832a13fc426e843ddffa5fe8facb P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:43.868631 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet fd69832a13fc426e843ddffa5fe8facb (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:43.869029 17480 tablet_replica.cc:331] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:43.869261 17480 raft_consensus.cc:2238] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:43.869459 17480 raft_consensus.cc:2267] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:43.870455 17480 ts_tablet_manager.cc:1905] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:43.873591 17507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:43.877961 17475 tablet_bootstrap.cc:492] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:43.878041 17453 fs_manager.cc:730] Time spent opening block manager: real 0.038s	user 0.012s	sys 0.003s
I20250111 08:58:43.878120 17475 ts_tablet_manager.cc:1397] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.031s	user 0.009s	sys 0.002s
I20250111 08:58:43.878964 17475 raft_consensus.cc:357] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.879165 17475 raft_consensus.cc:383] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.879254 17475 raft_consensus.cc:738] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.878225 17453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:43.879453 17453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:43.885349 17471 tablet_bootstrap.cc:492] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:43.885531 17471 ts_tablet_manager.cc:1397] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.032s	user 0.004s	sys 0.006s
I20250111 08:58:43.886416 17471 raft_consensus.cc:357] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.886653 17471 raft_consensus.cc:383] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.886777 17471 raft_consensus.cc:738] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.887081 17471 consensus_queue.cc:260] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.893184 17471 ts_tablet_manager.cc:1428] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.001s
I20250111 08:58:43.898041 17471 tablet_bootstrap.cc:492] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:43.899423 17480 ts_tablet_manager.cc:1918] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:43.899763 17471 tablet_bootstrap.cc:654] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.901024 17480 log.cc:1198] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/cca218fd9cb84cb796bdea731785f440
I20250111 08:58:43.900534 17475 consensus_queue.cc:260] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.901633 17475 ts_tablet_manager.cc:1428] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.023s	user 0.001s	sys 0.001s
I20250111 08:58:43.902698 17475 tablet_bootstrap.cc:492] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:43.903834 17474 tablet_bootstrap.cc:492] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:43.904048 17474 ts_tablet_manager.cc:1397] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.080s	user 0.009s	sys 0.009s
I20250111 08:58:43.904333 17475 tablet_bootstrap.cc:654] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.905015 17474 raft_consensus.cc:357] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.905295 17474 raft_consensus.cc:383] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.905476 17474 raft_consensus.cc:738] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.905782 17474 consensus_queue.cc:260] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.908743 17474 ts_tablet_manager.cc:1428] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20250111 08:58:43.910425 16877 catalog_manager.cc:2149] created HMS catalog entry for table  [id=1620935ff43440ec8c2fb76207fce81d]
W20250111 08:58:43.911275 16877 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER
I20250111 08:58:43.912068 17474 tablet_bootstrap.cc:492] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:43.914076 17474 tablet_bootstrap.cc:654] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:43.914309 17480 ts_tablet_manager.cc:1939] T cca218fd9cb84cb796bdea731785f440 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:43.915424 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet cca218fd9cb84cb796bdea731785f440 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:43.921584 17474 tablet_bootstrap.cc:492] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:43.922407 17474 ts_tablet_manager.cc:1397] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.000s
I20250111 08:58:43.922755 17475 tablet_bootstrap.cc:492] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:43.922921 17475 ts_tablet_manager.cc:1397] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.005s
I20250111 08:58:43.923592 17475 raft_consensus.cc:357] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.923576 17474 raft_consensus.cc:357] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.923794 17475 raft_consensus.cc:383] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.923867 17475 raft_consensus.cc:738] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.924151 17475 consensus_queue.cc:260] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.924877 17474 raft_consensus.cc:383] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.925137 17474 raft_consensus.cc:738] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.925669 17475 ts_tablet_manager.cc:1428] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20250111 08:58:43.925767 17474 consensus_queue.cc:260] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.928774 17474 ts_tablet_manager.cc:1428] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.000s
I20250111 08:58:43.929797 17486 raft_consensus.cc:491] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:43.929953 17486 raft_consensus.cc:513] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.930589 17486 leader_election.cc:290] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:43.931067 17471 tablet_bootstrap.cc:492] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:43.931221 17471 ts_tablet_manager.cc:1397] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.033s	user 0.006s	sys 0.000s
I20250111 08:58:43.931264 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd28cf5c804d88b7877d57a1cc2248" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7" is_pre_election: true
I20250111 08:58:43.931478 17209 raft_consensus.cc:2463] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 0.
I20250111 08:58:43.931949 17006 leader_election.cc:304] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7; no voters: 
I20250111 08:58:43.931910 17471 raft_consensus.cc:357] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.932102 17471 raft_consensus.cc:383] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:43.932155 17471 raft_consensus.cc:738] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:43.932384 17471 consensus_queue.cc:260] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.932858 17471 ts_tablet_manager.cc:1428] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20250111 08:58:43.933066 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd28cf5c804d88b7877d57a1cc2248" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe" is_pre_election: true
I20250111 08:58:43.933281 17343 raft_consensus.cc:2463] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 0.
I20250111 08:58:43.934759 17486 raft_consensus.cc:2798] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 08:58:43.934846 17486 raft_consensus.cc:491] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:58:43.934926 17486 raft_consensus.cc:3054] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:43.937358 17486 raft_consensus.cc:513] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.937960 17486 leader_election.cc:290] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 election: Requested vote from peers d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:43.938411 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd28cf5c804d88b7877d57a1cc2248" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7"
I20250111 08:58:43.938625 17209 raft_consensus.cc:3054] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:43.941018 17453 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:43.941120 17209 raft_consensus.cc:2463] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 1.
I20250111 08:58:43.941663 17006 leader_election.cc:304] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7; no voters: 
I20250111 08:58:43.941668 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acfd28cf5c804d88b7877d57a1cc2248" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe"
I20250111 08:58:43.941881 17343 raft_consensus.cc:3054] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:43.941996 17486 raft_consensus.cc:2798] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 08:58:43.942072 17453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:43.942361 17453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:43.943296 17486 raft_consensus.cc:695] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 LEADER]: Becoming Leader. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Running, Role: LEADER
I20250111 08:58:43.943630 17486 consensus_queue.cc:237] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:43.944407 17343 raft_consensus.cc:2463] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 1.
I20250111 08:58:43.950628 16733 catalog_manager.cc:5526] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129). New cstate: current_term: 1 leader_uuid: "13afd9f6a62a4efd831797a347bb31b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } health_report { overall_health: UNKNOWN } } }
W20250111 08:58:43.978650 17257 tablet.cc:2367] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 08:58:44.001695 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet d58685698a294cbe8834cdfcb4d14548 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.007495 16731 master_service.cc:946] ReplaceTablet: received request to replace tablet acfd28cf5c804d88b7877d57a1cc2248 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.015252 16735 master_service.cc:946] ReplaceTablet: received request to replace tablet acfd28cf5c804d88b7877d57a1cc2248 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.015337 16734 master_service.cc:946] ReplaceTablet: received request to replace tablet d58685698a294cbe8834cdfcb4d14548 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.016324 16733 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet d58685698a294cbe8834cdfcb4d14548
I20250111 08:58:44.017042 16733 catalog_manager.cc:6596] ReplaceTablet: tablet d58685698a294cbe8834cdfcb4d14548 deleted and replaced by tablet b1ccca4bdde24f9c95b1d36145f7b534
I20250111 08:58:44.019011 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet d58685698a294cbe8834cdfcb4d14548 with delete_type TABLET_DATA_DELETED (replaced by tablet b1ccca4bdde24f9c95b1d36145f7b534) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.019326 17480 tablet_replica.cc:331] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:44.019520 17480 raft_consensus.cc:2238] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.019722 17480 raft_consensus.cc:2267] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.020656 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet d58685698a294cbe8834cdfcb4d14548 with delete_type TABLET_DATA_DELETED (replaced by tablet b1ccca4bdde24f9c95b1d36145f7b534) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.020924 17481 tablet_replica.cc:331] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:44.021063 16874 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=3f0673839c43477080a94acfef4b5125]
I20250111 08:58:44.021099 17481 raft_consensus.cc:2238] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.021255 17481 raft_consensus.cc:2267] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.021353 16874 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 5) took 1753 ms. Trace:
I20250111 08:58:44.021464 16874 rpcz_store.cc:276] 0111 08:58:42.267445 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.267509 (+    64us) service_pool.cc:225] Handling call
0111 08:58:42.704499 (+436990us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.704937 (+   438us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.339696 (+634759us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.340716 (+  1020us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.340720 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:43.340971 (+   251us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.021316 (+680345us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.340620 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.340634 (+    14us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.340642 (+     8us) write_op.cc:454] Released schema lock
0111 08:58:43.340647 (+     5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1153451,"Hive Metastore.run_cpu_time_us":532,"Hive Metastore.run_wall_time_us":161052,"child_traces":[["op",{}]]}
I20250111 08:58:44.024077 17481 ts_tablet_manager.cc:1905] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.029014 17480 ts_tablet_manager.cc:1905] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.037150 16731 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet acfd28cf5c804d88b7877d57a1cc2248
I20250111 08:58:44.037667 16731 catalog_manager.cc:6596] ReplaceTablet: tablet acfd28cf5c804d88b7877d57a1cc2248 deleted and replaced by tablet 699ec279a76343d189475aca41b4b6e6
I20250111 08:58:44.038300 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet acfd28cf5c804d88b7877d57a1cc2248 with delete_type TABLET_DATA_DELETED (replaced by tablet 699ec279a76343d189475aca41b4b6e6) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.038940 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet acfd28cf5c804d88b7877d57a1cc2248 with delete_type TABLET_DATA_DELETED (replaced by tablet 699ec279a76343d189475aca41b4b6e6) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.041582 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet d58685698a294cbe8834cdfcb4d14548 with delete_type TABLET_DATA_DELETED (replaced by tablet b1ccca4bdde24f9c95b1d36145f7b534) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.041792 17188 tablet_service.cc:1514] Processing DeleteTablet for tablet acfd28cf5c804d88b7877d57a1cc2248 with delete_type TABLET_DATA_DELETED (replaced by tablet 699ec279a76343d189475aca41b4b6e6) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.041898 17482 tablet_replica.cc:331] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:44.042104 17482 raft_consensus.cc:2238] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.042287 17482 raft_consensus.cc:2267] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.043036 17482 ts_tablet_manager.cc:1905] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.049074 17482 ts_tablet_manager.cc:1918] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.049285 17482 log.cc:1198] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/d58685698a294cbe8834cdfcb4d14548
I20250111 08:58:44.049674 17482 ts_tablet_manager.cc:1939] T d58685698a294cbe8834cdfcb4d14548 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:44.050158 17051 tablet_service.cc:1467] Processing CreateTablet for tablet b1ccca4bdde24f9c95b1d36145f7b534 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.050570 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ccca4bdde24f9c95b1d36145f7b534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.050726 17482 tablet_replica.cc:331] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:44.050940 17482 raft_consensus.cc:2238] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.051172 17482 raft_consensus.cc:2267] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.051923 17482 ts_tablet_manager.cc:1905] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.053071 17323 tablet_service.cc:1467] Processing CreateTablet for tablet b1ccca4bdde24f9c95b1d36145f7b534 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.053481 17323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ccca4bdde24f9c95b1d36145f7b534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.057420 17482 ts_tablet_manager.cc:1918] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.058260 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet d58685698a294cbe8834cdfcb4d14548 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.058717 16731 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_7d1113f20be344b0802727c6883a0652"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:44.059022 16735 master_service.cc:946] ReplaceTablet: received request to replace tablet d58685698a294cbe8834cdfcb4d14548 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.062237 17482 log.cc:1198] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/acfd28cf5c804d88b7877d57a1cc2248
I20250111 08:58:44.062813 17482 ts_tablet_manager.cc:1939] T acfd28cf5c804d88b7877d57a1cc2248 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:44.064182 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet acfd28cf5c804d88b7877d57a1cc2248 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:44.064646 17188 tablet_service.cc:1467] Processing CreateTablet for tablet b1ccca4bdde24f9c95b1d36145f7b534 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.065029 17188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ccca4bdde24f9c95b1d36145f7b534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.065711 17050 tablet_service.cc:1467] Processing CreateTablet for tablet 699ec279a76343d189475aca41b4b6e6 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.066071 17050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699ec279a76343d189475aca41b4b6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.069250 17189 tablet_service.cc:1467] Processing CreateTablet for tablet 699ec279a76343d189475aca41b4b6e6 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.069577 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699ec279a76343d189475aca41b4b6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.070786 17474 tablet_bootstrap.cc:492] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.072511 17474 tablet_bootstrap.cc:654] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.073346 17324 tablet_service.cc:1467] Processing CreateTablet for tablet 699ec279a76343d189475aca41b4b6e6 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.073763 17324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699ec279a76343d189475aca41b4b6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.084064 17471 tablet_bootstrap.cc:492] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.085806 17471 tablet_bootstrap.cc:654] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.086902 17474 tablet_bootstrap.cc:492] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.087098 17474 ts_tablet_manager.cc:1397] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.003s
I20250111 08:58:44.087877 17474 raft_consensus.cc:357] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.088141 17474 raft_consensus.cc:383] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.088261 17474 raft_consensus.cc:738] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.088559 17474 consensus_queue.cc:260] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.091717 17474 ts_tablet_manager.cc:1428] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20250111 08:58:44.094975 17474 tablet_bootstrap.cc:492] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.097142 17474 tablet_bootstrap.cc:654] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.098557 17480 ts_tablet_manager.cc:1918] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.098695 17480 log.cc:1198] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/d58685698a294cbe8834cdfcb4d14548
I20250111 08:58:44.099134 17480 ts_tablet_manager.cc:1939] T d58685698a294cbe8834cdfcb4d14548 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:44.100157 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet d58685698a294cbe8834cdfcb4d14548 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.100561 17480 tablet_replica.cc:331] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:44.102671 17481 ts_tablet_manager.cc:1918] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.102891 17481 log.cc:1198] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/d58685698a294cbe8834cdfcb4d14548
I20250111 08:58:44.103314 17481 ts_tablet_manager.cc:1939] T d58685698a294cbe8834cdfcb4d14548 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:44.104288 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet d58685698a294cbe8834cdfcb4d14548 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.104748 17481 tablet_replica.cc:331] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:44.104960 17481 raft_consensus.cc:2238] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.105177 17481 raft_consensus.cc:2267] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.105968 17481 ts_tablet_manager.cc:1905] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.106299 17475 tablet_bootstrap.cc:492] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.109043 17475 tablet_bootstrap.cc:654] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.110087 17474 tablet_bootstrap.cc:492] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.110244 17474 ts_tablet_manager.cc:1397] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.003s
I20250111 08:58:44.111014 17474 raft_consensus.cc:357] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.111238 17474 raft_consensus.cc:383] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.111308 17474 raft_consensus.cc:738] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.111570 17474 consensus_queue.cc:260] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.114725 17474 ts_tablet_manager.cc:1428] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20250111 08:58:44.114746 17480 raft_consensus.cc:2238] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 LEADER]: Raft consensus shutting down.
I20250111 08:58:44.115454 17480 pending_rounds.cc:62] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Trying to abort 1 pending ops.
I20250111 08:58:44.121546 17480 pending_rounds.cc:69] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113055944476352512 op_type: NO_OP noop_request { }
I20250111 08:58:44.122364 17480 raft_consensus.cc:2883] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 08:58:44.123289 17480 raft_consensus.cc:2267] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.124614 17480 ts_tablet_manager.cc:1905] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.136308 16876 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=eb5afce7057a493aae25a704cda87ea4]
I20250111 08:58:44.136765 17475 tablet_bootstrap.cc:492] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.136674 16876 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 3) took 1877 ms. Trace:
I20250111 08:58:44.136907 17475 ts_tablet_manager.cc:1397] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.031s	user 0.005s	sys 0.004s
I20250111 08:58:44.136976 16876 rpcz_store.cc:276] 0111 08:58:42.258766 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.259136 (+   370us) service_pool.cc:225] Handling call
0111 08:58:42.704292 (+445156us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.704903 (+   611us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.401673 (+696770us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.402406 (+   733us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.402410 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:43.402600 (+   190us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.136631 (+734031us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.402343 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.402352 (+     9us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.402357 (+     5us) write_op.cc:454] Released schema lock
0111 08:58:43.402360 (+     3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1253000,"Hive Metastore.run_cpu_time_us":471,"Hive Metastore.run_wall_time_us":176822,"mutex_wait_us":29,"child_traces":[["op",{}]]}
I20250111 08:58:44.140781 17475 raft_consensus.cc:357] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.140986 17475 raft_consensus.cc:383] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.141076 17475 raft_consensus.cc:738] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.141650 16735 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:44.141330 17475 consensus_queue.cc:260] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.145551 17481 ts_tablet_manager.cc:1918] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.145689 17481 log.cc:1198] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/acfd28cf5c804d88b7877d57a1cc2248
I20250111 08:58:44.145768 17475 ts_tablet_manager.cc:1428] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.002s
I20250111 08:58:44.146075 17481 ts_tablet_manager.cc:1939] T acfd28cf5c804d88b7877d57a1cc2248 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:44.147116 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet acfd28cf5c804d88b7877d57a1cc2248 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:44.150919 17475 tablet_bootstrap.cc:492] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.151808 17480 ts_tablet_manager.cc:1918] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 08:58:44.152565 17475 tablet_bootstrap.cc:654] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.152587 17471 tablet_bootstrap.cc:492] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.152835 17471 ts_tablet_manager.cc:1397] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.069s	user 0.009s	sys 0.019s
I20250111 08:58:44.153664 17471 raft_consensus.cc:357] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.153944 17471 raft_consensus.cc:383] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.154065 17471 raft_consensus.cc:738] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.154372 16738 catalog_manager.cc:3422] renamed table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6] in HMS: new name default.table_4838c339e9c747e1b906df07efeba1bc
I20250111 08:58:44.154320 17471 consensus_queue.cc:260] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.158007 17471 ts_tablet_manager.cc:1428] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.001s
I20250111 08:58:44.161536 17471 tablet_bootstrap.cc:492] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.163244 17471 tablet_bootstrap.cc:654] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.164129 17480 log.cc:1198] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/acfd28cf5c804d88b7877d57a1cc2248
I20250111 08:58:44.164635 17480 ts_tablet_manager.cc:1939] T acfd28cf5c804d88b7877d57a1cc2248 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:44.168536 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet acfd28cf5c804d88b7877d57a1cc2248 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:44.177556 17475 tablet_bootstrap.cc:492] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.177742 17475 ts_tablet_manager.cc:1397] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.027s	user 0.002s	sys 0.004s
I20250111 08:58:44.178575 17475 raft_consensus.cc:357] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.178819 17475 raft_consensus.cc:383] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.178905 17475 raft_consensus.cc:738] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.179152 17475 consensus_queue.cc:260] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.179692 17475 ts_tablet_manager.cc:1428] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20250111 08:58:44.180832 17471 tablet_bootstrap.cc:492] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.180985 17471 ts_tablet_manager.cc:1397] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.020s	user 0.001s	sys 0.002s
I20250111 08:58:44.181753 17471 raft_consensus.cc:357] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.181967 17471 raft_consensus.cc:383] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.182036 17471 raft_consensus.cc:738] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.182263 17471 consensus_queue.cc:260] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.197232 17471 ts_tablet_manager.cc:1428] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.000s
I20250111 08:58:44.239953 16734 master_service.cc:946] ReplaceTablet: received request to replace tablet 699ec279a76343d189475aca41b4b6e6 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.244619 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet 699ec279a76343d189475aca41b4b6e6 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.247659 16730 master_service.cc:946] ReplaceTablet: received request to replace tablet b1ccca4bdde24f9c95b1d36145f7b534 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.248466 16729 master_service.cc:946] ReplaceTablet: received request to replace tablet 699ec279a76343d189475aca41b4b6e6 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.249622 16734 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet 699ec279a76343d189475aca41b4b6e6
I20250111 08:58:44.250123 16734 catalog_manager.cc:6596] ReplaceTablet: tablet 699ec279a76343d189475aca41b4b6e6 deleted and replaced by tablet a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:58:44.251056 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet 699ec279a76343d189475aca41b4b6e6 with delete_type TABLET_DATA_DELETED (replaced by tablet a2f683bacb4e4778b7ef52d75fdb0832) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.251546 17480 tablet_replica.cc:331] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:44.251526 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet 699ec279a76343d189475aca41b4b6e6 with delete_type TABLET_DATA_DELETED (replaced by tablet a2f683bacb4e4778b7ef52d75fdb0832) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.251770 17480 raft_consensus.cc:2238] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.251796 17482 tablet_replica.cc:331] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:44.251967 17480 raft_consensus.cc:2267] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.252002 17482 raft_consensus.cc:2238] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.252187 17482 raft_consensus.cc:2267] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.253258 17482 ts_tablet_manager.cc:1905] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.254482 17480 ts_tablet_manager.cc:1905] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.256541 17324 tablet_service.cc:1514] Processing DeleteTablet for tablet 699ec279a76343d189475aca41b4b6e6 with delete_type TABLET_DATA_DELETED (replaced by tablet a2f683bacb4e4778b7ef52d75fdb0832) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.256820 17481 tablet_replica.cc:331] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:44.257022 17481 raft_consensus.cc:2238] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.257193 17481 raft_consensus.cc:2267] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.257794 16737 catalog_manager.cc:3422] renamed table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a] in HMS: new name default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
I20250111 08:58:44.258409 17481 ts_tablet_manager.cc:1905] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.259559 16730 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet b1ccca4bdde24f9c95b1d36145f7b534
I20250111 08:58:44.260033 16730 catalog_manager.cc:6596] ReplaceTablet: tablet b1ccca4bdde24f9c95b1d36145f7b534 deleted and replaced by tablet 8ca8a1d44019417d82327374a2ec2574
I20250111 08:58:44.261346 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ccca4bdde24f9c95b1d36145f7b534 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ca8a1d44019417d82327374a2ec2574) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.262074 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ccca4bdde24f9c95b1d36145f7b534 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ca8a1d44019417d82327374a2ec2574) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.262311 17324 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ccca4bdde24f9c95b1d36145f7b534 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ca8a1d44019417d82327374a2ec2574) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.267889 17482 ts_tablet_manager.cc:1918] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.268035 17482 log.cc:1198] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/699ec279a76343d189475aca41b4b6e6
I20250111 08:58:44.269145 16730 master_service.cc:946] ReplaceTablet: received request to replace tablet 699ec279a76343d189475aca41b4b6e6 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.270254 17481 ts_tablet_manager.cc:1918] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.270406 17481 log.cc:1198] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/699ec279a76343d189475aca41b4b6e6
I20250111 08:58:44.270802 17481 ts_tablet_manager.cc:1939] T 699ec279a76343d189475aca41b4b6e6 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:44.271611 17189 tablet_service.cc:1467] Processing CreateTablet for tablet a2f683bacb4e4778b7ef52d75fdb0832 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.271993 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2f683bacb4e4778b7ef52d75fdb0832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.275875 17324 tablet_service.cc:1467] Processing CreateTablet for tablet a2f683bacb4e4778b7ef52d75fdb0832 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.276250 17324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2f683bacb4e4778b7ef52d75fdb0832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.280850 17051 tablet_service.cc:1467] Processing CreateTablet for tablet a2f683bacb4e4778b7ef52d75fdb0832 (DEFAULT_TABLE table=default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.281260 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2f683bacb4e4778b7ef52d75fdb0832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.286664 17474 tablet_bootstrap.cc:492] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.287974 17480 ts_tablet_manager.cc:1918] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.288247 17480 log.cc:1198] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/699ec279a76343d189475aca41b4b6e6
I20250111 08:58:44.288252 17474 tablet_bootstrap.cc:654] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.289819 17475 tablet_bootstrap.cc:492] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.291467 17475 tablet_bootstrap.cc:654] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.291543 17471 tablet_bootstrap.cc:492] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.292156 17482 ts_tablet_manager.cc:1939] T 699ec279a76343d189475aca41b4b6e6 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:44.293143 17471 tablet_bootstrap.cc:654] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.293143 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet 699ec279a76343d189475aca41b4b6e6 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:44.293908 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet 699ec279a76343d189475aca41b4b6e6 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:44.294206 17481 tablet_replica.cc:331] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:44.294373 17481 raft_consensus.cc:2238] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.294541 17481 raft_consensus.cc:2267] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.294855 17482 tablet_replica.cc:331] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:44.295018 17482 raft_consensus.cc:2238] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.295178 17482 raft_consensus.cc:2267] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.295234 17481 ts_tablet_manager.cc:1905] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.300084 17480 ts_tablet_manager.cc:1939] T 699ec279a76343d189475aca41b4b6e6 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:44.301311 17481 ts_tablet_manager.cc:1918] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.301431 17481 log.cc:1198] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/b1ccca4bdde24f9c95b1d36145f7b534
I20250111 08:58:44.301800 17481 ts_tablet_manager.cc:1939] T b1ccca4bdde24f9c95b1d36145f7b534 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:44.302708 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet b1ccca4bdde24f9c95b1d36145f7b534 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.302955 17482 ts_tablet_manager.cc:1905] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.303611 17480 tablet_replica.cc:331] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:44.303794 17480 raft_consensus.cc:2238] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.303984 17480 raft_consensus.cc:2267] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.304756 17480 ts_tablet_manager.cc:1905] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.328471 16871 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=f00998fbd79b4a5baec97891252d6a98]
I20250111 08:58:44.328595 17482 ts_tablet_manager.cc:1918] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.328799 17482 log.cc:1198] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/b1ccca4bdde24f9c95b1d36145f7b534
I20250111 08:58:44.328827 16871 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 8) took 2061 ms. Trace:
I20250111 08:58:44.329159 17482 ts_tablet_manager.cc:1939] T b1ccca4bdde24f9c95b1d36145f7b534 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:44.328956 16871 rpcz_store.cc:276] 0111 08:58:42.267789 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.268402 (+   613us) service_pool.cc:225] Handling call
0111 08:58:42.704587 (+436185us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.705112 (+   525us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.483805 (+778693us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.484727 (+   922us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.484731 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:43.484957 (+   226us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.328792 (+843835us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.484647 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.484659 (+    12us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.484668 (+     9us) write_op.cc:454] Released schema lock
0111 08:58:43.484673 (+     5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1347309,"Hive Metastore.run_cpu_time_us":463,"Hive Metastore.run_wall_time_us":274165,"child_traces":[["op",{}]]}
W20250111 08:58:44.330698 17398 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.190:46487)
I20250111 08:58:44.334528 16730 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:44.336591 17324 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca8a1d44019417d82327374a2ec2574 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.337859 17189 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca8a1d44019417d82327374a2ec2574 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.338294 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca8a1d44019417d82327374a2ec2574. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.340509 17324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca8a1d44019417d82327374a2ec2574. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.345163 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet b1ccca4bdde24f9c95b1d36145f7b534 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.355161 17475 tablet_bootstrap.cc:492] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.355350 17475 ts_tablet_manager.cc:1397] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.066s	user 0.011s	sys 0.001s
I20250111 08:58:44.356238 17475 raft_consensus.cc:357] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.356482 17475 raft_consensus.cc:383] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.356576 17475 raft_consensus.cc:738] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.356869 17475 consensus_queue.cc:260] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.357429 17475 ts_tablet_manager.cc:1428] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20250111 08:58:44.361047 17471 tablet_bootstrap.cc:492] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.361212 17471 ts_tablet_manager.cc:1397] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.070s	user 0.008s	sys 0.016s
I20250111 08:58:44.362025 17471 raft_consensus.cc:357] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.362246 17471 raft_consensus.cc:383] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.362335 17471 raft_consensus.cc:738] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.362589 17471 consensus_queue.cc:260] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.363121 17471 ts_tablet_manager.cc:1428] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20250111 08:58:44.363406 17471 tablet_bootstrap.cc:492] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.365877 17471 tablet_bootstrap.cc:654] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.368511 17474 tablet_bootstrap.cc:492] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.368680 17474 ts_tablet_manager.cc:1397] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.082s	user 0.014s	sys 0.020s
I20250111 08:58:44.369421 17474 raft_consensus.cc:357] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.369635 17474 raft_consensus.cc:383] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.369714 17474 raft_consensus.cc:738] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.369973 17474 consensus_queue.cc:260] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.370504 17474 ts_tablet_manager.cc:1428] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20250111 08:58:44.370803 17474 tablet_bootstrap.cc:492] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.373454 17474 tablet_bootstrap.cc:654] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.393728 17474 tablet_bootstrap.cc:492] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.393939 17474 ts_tablet_manager.cc:1397] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.016s
I20250111 08:58:44.395553 17474 raft_consensus.cc:357] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.395814 17474 raft_consensus.cc:383] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.395877 17474 raft_consensus.cc:738] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.396145 17474 consensus_queue.cc:260] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.396710 17474 ts_tablet_manager.cc:1428] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
W20250111 08:58:44.400550 16724 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.20.189:35063, user_credentials={real_user=slave}} blocked reactor thread for 79530.8us
I20250111 08:58:44.401517 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet 699ec279a76343d189475aca41b4b6e6 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]) successfully deleted
I20250111 08:58:44.402817 17051 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca8a1d44019417d82327374a2ec2574 (DEFAULT_TABLE table=default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.403237 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca8a1d44019417d82327374a2ec2574. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.403298 17471 tablet_bootstrap.cc:492] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.403470 17471 ts_tablet_manager.cc:1397] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.040s	user 0.003s	sys 0.000s
I20250111 08:58:44.404220 17471 raft_consensus.cc:357] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.404453 17471 raft_consensus.cc:383] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.404532 17471 raft_consensus.cc:738] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.404789 17471 consensus_queue.cc:260] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.405323 17471 ts_tablet_manager.cc:1428] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20250111 08:58:44.408236 16732 catalog_manager.cc:2149] created HMS catalog entry for table  [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:58:44.416483 16732 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34578:
table { table_name: "default.Table_4b95455ed62a4f51a8aa85b327bc4699" } new_table_name: "default.Table_339512fd53274d4a9271fe2ca434c6c7" modify_external_catalogs: true
I20250111 08:58:44.419020 17475 tablet_bootstrap.cc:492] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.420922 17475 tablet_bootstrap.cc:654] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.423904 17475 tablet_bootstrap.cc:492] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.424775 17475 ts_tablet_manager.cc:1397] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20250111 08:58:44.426316 17324 tablet_service.cc:1467] Processing CreateTablet for tablet 163857e9dd0b446ba48262c908e551e7 (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.426789 17324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163857e9dd0b446ba48262c908e551e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.427803 16959 catalog_manager.cc:2710] Deleting table default.table_7d1113f20be344b0802727c6883a0652 [id=3f0673839c43477080a94acfef4b5125] in response to Hive Metastore notification log event 15
W20250111 08:58:44.428086 16959 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 15 DROP_TABLE default.table_7d1113f20be344b0802727c6883a0652: Not found: the table does not exist: table_id: "3f0673839c43477080a94acfef4b5125" table_name: "default.table_7d1113f20be344b0802727c6883a0652"
I20250111 08:58:44.431143 17480 ts_tablet_manager.cc:1918] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.431277 17480 log.cc:1198] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/b1ccca4bdde24f9c95b1d36145f7b534
I20250111 08:58:44.431684 17480 ts_tablet_manager.cc:1939] T b1ccca4bdde24f9c95b1d36145f7b534 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:44.432778 17189 tablet_service.cc:1467] Processing CreateTablet for tablet 163857e9dd0b446ba48262c908e551e7 (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.433369 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet b1ccca4bdde24f9c95b1d36145f7b534 (table default.table_bd323ab641234f2ebfe5a90526f93cdd [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.434208 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163857e9dd0b446ba48262c908e551e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.440044 17475 raft_consensus.cc:357] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.440310 17475 raft_consensus.cc:383] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.440404 17475 raft_consensus.cc:738] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
W20250111 08:58:44.440348 16959 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ca8a1d44019417d82327374a2ec2574 consensus state has no leader
F20250111 08:58:44.440799 16959 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 16): 16 DROP_TABLE default.table_518dba2592a54eddbd2b2caec0bb4cfa
I20250111 08:58:44.440778 17475 consensus_queue.cc:260] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
*** Check failure stack trace: ***
    @     0x7ff223737dcd  google::LogMessage::Fail() at ??:0
I20250111 08:58:44.445358 17475 ts_tablet_manager.cc:1428] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.020s	user 0.000s	sys 0.002s
I20250111 08:58:44.445559 16729 catalog_manager.cc:5618] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699) does not have the latest schema for tablet 8ca8a1d44019417d82327374a2ec2574 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]). Expected version 1 got 0
    @     0x7ff22373bb93  google::LogMessage::SendToLog() at ??:0
    @     0x7ff2237377cc  google::LogMessage::Flush() at ??:0
    @     0x7ff223737a11  google::LogMessage::~LogMessage() at ??:0
I20250111 08:58:44.451743 16729 catalog_manager.cc:5618] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699) does not have the latest schema for tablet 8ca8a1d44019417d82327374a2ec2574 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]). Expected version 1 got 0
I20250111 08:58:44.452651 16729 master_service.cc:946] ReplaceTablet: received request to replace tablet 163857e9dd0b446ba48262c908e551e7 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.453280 17471 tablet_bootstrap.cc:492] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.454927 16734 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ca8a1d44019417d82327374a2ec2574 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.454975 17471 tablet_bootstrap.cc:654] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.457806 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet 163857e9dd0b446ba48262c908e551e7 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.458365 17051 tablet_service.cc:1467] Processing CreateTablet for tablet 163857e9dd0b446ba48262c908e551e7 (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.464037 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ca8a1d44019417d82327374a2ec2574 from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.464218 16734 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet 8ca8a1d44019417d82327374a2ec2574
I20250111 08:58:44.464735 16734 catalog_manager.cc:6596] ReplaceTablet: tablet 8ca8a1d44019417d82327374a2ec2574 deleted and replaced by tablet 6634d4c9c6e846ae8c655b59f8d001d8
I20250111 08:58:44.464758 16729 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet 163857e9dd0b446ba48262c908e551e7
I20250111 08:58:44.465164 16729 catalog_manager.cc:6596] ReplaceTablet: tablet 163857e9dd0b446ba48262c908e551e7 deleted and replaced by tablet 3586e89f11214f869d9922fe7d8a5bca
I20250111 08:58:44.465765 17051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163857e9dd0b446ba48262c908e551e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.467540 17323 tablet_service.cc:1514] Processing DeleteTablet for tablet 163857e9dd0b446ba48262c908e551e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3586e89f11214f869d9922fe7d8a5bca) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.468230 17050 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca8a1d44019417d82327374a2ec2574 with delete_type TABLET_DATA_DELETED (replaced by tablet 6634d4c9c6e846ae8c655b59f8d001d8) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.468560 17049 tablet_service.cc:1514] Processing DeleteTablet for tablet 163857e9dd0b446ba48262c908e551e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3586e89f11214f869d9922fe7d8a5bca) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.468868 17480 tablet_replica.cc:331] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:44.469061 17480 raft_consensus.cc:2238] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.469241 17480 raft_consensus.cc:2267] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250111 08:58:44.469285 16723 catalog_manager.cc:4852] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): delete failed for tablet 163857e9dd0b446ba48262c908e551e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163857e9dd0b446ba48262c908e551e7 already in progress: creating tablet
I20250111 08:58:44.471177 17474 tablet_bootstrap.cc:492] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.472904 17474 tablet_bootstrap.cc:654] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.473549 16872 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=30e8d8c675d1451d9f2772e635af7bd4]
I20250111 08:58:44.473841 16872 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 7) took 2206 ms. Trace:
I20250111 08:58:44.473939 16872 rpcz_store.cc:276] 0111 08:58:42.267679 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.268296 (+   617us) service_pool.cc:225] Handling call
0111 08:58:42.704796 (+436500us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.705202 (+   406us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.627195 (+921993us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.627950 (+   755us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.627953 (+     3us) write_op.cc:454] Released schema lock
0111 08:58:43.628142 (+   189us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.473811 (+845669us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.627879 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.627891 (+    12us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.627898 (+     7us) write_op.cc:454] Released schema lock
0111 08:58:43.627903 (+     5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1478366,"Hive Metastore.run_cpu_time_us":497,"Hive Metastore.run_wall_time_us":288440,"child_traces":[["op",{}]]}
I20250111 08:58:44.475347 17480 ts_tablet_manager.cc:1905] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.475987 17049 tablet_service.cc:1467] Processing CreateTablet for tablet 3586e89f11214f869d9922fe7d8a5bca (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.476548 17049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3586e89f11214f869d9922fe7d8a5bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.480701 17050 tablet_service.cc:1467] Processing CreateTablet for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (DEFAULT_TABLE table=default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.481078 17050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6634d4c9c6e846ae8c655b59f8d001d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.482012 17480 ts_tablet_manager.cc:1918] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.482156 17480 log.cc:1198] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/8ca8a1d44019417d82327374a2ec2574
I20250111 08:58:44.482508 17480 ts_tablet_manager.cc:1939] T 8ca8a1d44019417d82327374a2ec2574 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:44.484830 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet 8ca8a1d44019417d82327374a2ec2574 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
W20250111 08:58:44.485175 16724 catalog_manager.cc:4839] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): delete failed for tablet 163857e9dd0b446ba48262c908e551e7 because the tablet was not found. No further retry: Not found: Tablet not found: 163857e9dd0b446ba48262c908e551e7
I20250111 08:58:44.486171 17323 tablet_service.cc:1467] Processing CreateTablet for tablet 3586e89f11214f869d9922fe7d8a5bca (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.486570 17323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3586e89f11214f869d9922fe7d8a5bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.487030 17188 tablet_service.cc:1514] Processing DeleteTablet for tablet 163857e9dd0b446ba48262c908e551e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3586e89f11214f869d9922fe7d8a5bca) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.487063 17322 tablet_service.cc:1467] Processing CreateTablet for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (DEFAULT_TABLE table=default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.487509 17322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6634d4c9c6e846ae8c655b59f8d001d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.487566 17186 tablet_service.cc:1467] Processing CreateTablet for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (DEFAULT_TABLE table=default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250111 08:58:44.487768 16725 catalog_manager.cc:4852] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): delete failed for tablet 163857e9dd0b446ba48262c908e551e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163857e9dd0b446ba48262c908e551e7 already in progress: creating tablet
I20250111 08:58:44.487985 17186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6634d4c9c6e846ae8c655b59f8d001d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.492844 17187 tablet_service.cc:1467] Processing CreateTablet for tablet 3586e89f11214f869d9922fe7d8a5bca (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.493360 17187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3586e89f11214f869d9922fe7d8a5bca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 08:58:44.497516 16725 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8ca8a1d44019417d82327374a2ec2574 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8ca8a1d44019417d82327374a2ec2574 consensus state has no leader
I20250111 08:58:44.498728 17475 tablet_bootstrap.cc:492] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.500399 17475 tablet_bootstrap.cc:654] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.501832 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca8a1d44019417d82327374a2ec2574 with delete_type TABLET_DATA_DELETED (replaced by tablet 6634d4c9c6e846ae8c655b59f8d001d8) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.502108 17482 tablet_replica.cc:331] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:44.502301 17482 raft_consensus.cc:2238] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.502466 17482 raft_consensus.cc:2267] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.502620 17324 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca8a1d44019417d82327374a2ec2574 with delete_type TABLET_DATA_DELETED (replaced by tablet 6634d4c9c6e846ae8c655b59f8d001d8) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.502877 17481 tablet_replica.cc:331] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:44.503054 17481 raft_consensus.cc:2238] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.503202 17482 ts_tablet_manager.cc:1905] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.503226 17481 raft_consensus.cc:2267] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.503940 17481 ts_tablet_manager.cc:1905] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.509572 17481 ts_tablet_manager.cc:1918] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.509696 17481 log.cc:1198] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/8ca8a1d44019417d82327374a2ec2574
I20250111 08:58:44.510011 17481 ts_tablet_manager.cc:1939] T 8ca8a1d44019417d82327374a2ec2574 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:44.510175 17482 ts_tablet_manager.cc:1918] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.510293 17482 log.cc:1198] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/8ca8a1d44019417d82327374a2ec2574
I20250111 08:58:44.510610 17482 ts_tablet_manager.cc:1939] T 8ca8a1d44019417d82327374a2ec2574 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:44.510896 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet 8ca8a1d44019417d82327374a2ec2574 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.511718 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet 8ca8a1d44019417d82327374a2ec2574 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]) successfully deleted
I20250111 08:58:44.512344 16729 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34578:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
    @     0x7ff2398da6de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250111 08:58:44.516126 17475 tablet_bootstrap.cc:492] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.516263 17475 ts_tablet_manager.cc:1397] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.000s
I20250111 08:58:44.517081 17475 raft_consensus.cc:357] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.517287 17475 raft_consensus.cc:383] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.517362 17475 raft_consensus.cc:738] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.517587 17475 consensus_queue.cc:260] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.519623 16734 catalog_manager.cc:5618] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699) does not have the latest schema for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]). Expected version 1 got 0
W20250111 08:58:44.520190 16734 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6634d4c9c6e846ae8c655b59f8d001d8 consensus state has no leader
I20250111 08:58:44.521209 17475 ts_tablet_manager.cc:1428] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
    @     0x7ff2398d93c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250111 08:58:44.525326 16734 catalog_manager.cc:5618] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699) does not have the latest schema for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]). Expected version 1 got 0
I20250111 08:58:44.525624 16733 master_service.cc:946] ReplaceTablet: received request to replace tablet 3586e89f11214f869d9922fe7d8a5bca from {username='slave'} at 127.0.0.1:34578
I20250111 08:58:44.525457 17475 tablet_bootstrap.cc:492] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.528393 17475 tablet_bootstrap.cc:654] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.530187 17475 tablet_bootstrap.cc:492] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.530339 17475 ts_tablet_manager.cc:1397] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20250111 08:58:44.531098 17474 tablet_bootstrap.cc:492] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.531244 17474 ts_tablet_manager.cc:1397] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.060s	user 0.003s	sys 0.000s
I20250111 08:58:44.531142 17475 raft_consensus.cc:357] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.531349 17475 raft_consensus.cc:383] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.531415 17475 raft_consensus.cc:738] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.531415 16733 catalog_manager.cc:5813] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Sending DeleteTablet for 3 replicas of tablet 3586e89f11214f869d9922fe7d8a5bca
I20250111 08:58:44.531936 16733 catalog_manager.cc:6596] ReplaceTablet: tablet 3586e89f11214f869d9922fe7d8a5bca deleted and replaced by tablet b43238f36aa44f87b625287daef14f76
I20250111 08:58:44.531992 17474 raft_consensus.cc:357] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.532580 17471 tablet_bootstrap.cc:492] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.532725 17471 ts_tablet_manager.cc:1397] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.080s	user 0.006s	sys 0.027s
I20250111 08:58:44.532974 17474 raft_consensus.cc:383] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.533075 17474 raft_consensus.cc:738] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.533516 17471 raft_consensus.cc:357] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.533710 17471 raft_consensus.cc:383] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.533792 17322 tablet_service.cc:1514] Processing DeleteTablet for tablet 3586e89f11214f869d9922fe7d8a5bca with delete_type TABLET_DATA_DELETED (replaced by tablet b43238f36aa44f87b625287daef14f76) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.534003 17474 consensus_queue.cc:260] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.531630 17475 consensus_queue.cc:260] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.535609 17475 ts_tablet_manager.cc:1428] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20250111 08:58:44.535898 17475 tablet_bootstrap.cc:492] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.537760 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet 3586e89f11214f869d9922fe7d8a5bca with delete_type TABLET_DATA_DELETED (replaced by tablet b43238f36aa44f87b625287daef14f76) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.534475 17471 raft_consensus.cc:738] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.538161 17474 ts_tablet_manager.cc:1428] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20250111 08:58:44.538448 17474 tablet_bootstrap.cc:492] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.538620 17475 tablet_bootstrap.cc:654] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.539178 17471 consensus_queue.cc:260] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
W20250111 08:58:44.539686 16723 catalog_manager.cc:4852] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): delete failed for tablet 3586e89f11214f869d9922fe7d8a5bca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3586e89f11214f869d9922fe7d8a5bca already in progress: creating tablet
W20250111 08:58:44.539835 16725 catalog_manager.cc:4852] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): delete failed for tablet 3586e89f11214f869d9922fe7d8a5bca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3586e89f11214f869d9922fe7d8a5bca already in progress: creating tablet
I20250111 08:58:44.543848 17322 tablet_service.cc:1514] Processing DeleteTablet for tablet 163857e9dd0b446ba48262c908e551e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3586e89f11214f869d9922fe7d8a5bca) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.544013 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet 163857e9dd0b446ba48262c908e551e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3586e89f11214f869d9922fe7d8a5bca) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.544556 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet 3586e89f11214f869d9922fe7d8a5bca with delete_type TABLET_DATA_DELETED (replaced by tablet b43238f36aa44f87b625287daef14f76) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.544797 17480 tablet_replica.cc:331] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:44.544961 17480 raft_consensus.cc:2238] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.545123 17480 raft_consensus.cc:2267] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250111 08:58:44.545172 16723 catalog_manager.cc:4852] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): delete failed for tablet 163857e9dd0b446ba48262c908e551e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163857e9dd0b446ba48262c908e551e7 already in progress: creating tablet
I20250111 08:58:44.545697 17480 ts_tablet_manager.cc:1905] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.548004 17049 tablet_service.cc:1467] Processing CreateTablet for tablet b43238f36aa44f87b625287daef14f76 (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.548405 17049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43238f36aa44f87b625287daef14f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.548856 17474 tablet_bootstrap.cc:654] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.549744 17471 ts_tablet_manager.cc:1428] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.017s	user 0.002s	sys 0.000s
I20250111 08:58:44.551744 17480 ts_tablet_manager.cc:1918] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.551878 17480 log.cc:1198] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/3586e89f11214f869d9922fe7d8a5bca
I20250111 08:58:44.552291 17480 ts_tablet_manager.cc:1939] T 3586e89f11214f869d9922fe7d8a5bca P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:44.553412 16724 catalog_manager.cc:4872] TS 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): tablet 3586e89f11214f869d9922fe7d8a5bca (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]) successfully deleted
I20250111 08:58:44.554003 17482 tablet_replica.cc:331] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:44.554172 17482 raft_consensus.cc:2238] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.554373 17482 raft_consensus.cc:2267] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.554553 17471 tablet_bootstrap.cc:492] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.554788 17189 tablet_service.cc:1467] Processing CreateTablet for tablet b43238f36aa44f87b625287daef14f76 (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.555017 17322 tablet_service.cc:1467] Processing CreateTablet for tablet b43238f36aa44f87b625287daef14f76 (DEFAULT_TABLE table=default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 08:58:44.555150 17189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43238f36aa44f87b625287daef14f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.555241 17482 ts_tablet_manager.cc:1905] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.555369 17322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b43238f36aa44f87b625287daef14f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 08:58:44.556262 17471 tablet_bootstrap.cc:654] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.556768 17324 tablet_service.cc:1514] Processing DeleteTablet for tablet 163857e9dd0b446ba48262c908e551e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3586e89f11214f869d9922fe7d8a5bca) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.557049 17481 tablet_replica.cc:331] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:44.557222 17481 raft_consensus.cc:2238] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.557399 17481 raft_consensus.cc:2267] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.560832 17481 ts_tablet_manager.cc:1905] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.565397 17482 ts_tablet_manager.cc:1918] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.565512 17482 log.cc:1198] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/163857e9dd0b446ba48262c908e551e7
I20250111 08:58:44.565871 17482 ts_tablet_manager.cc:1939] T 163857e9dd0b446ba48262c908e551e7 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:44.566622 17471 tablet_bootstrap.cc:492] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.566784 17471 ts_tablet_manager.cc:1397] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.001s
I20250111 08:58:44.566812 17481 ts_tablet_manager.cc:1918] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.566794 16725 catalog_manager.cc:4872] TS d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): tablet 163857e9dd0b446ba48262c908e551e7 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]) successfully deleted
I20250111 08:58:44.566915 17481 log.cc:1198] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/163857e9dd0b446ba48262c908e551e7
I20250111 08:58:44.567265 17481 ts_tablet_manager.cc:1939] T 163857e9dd0b446ba48262c908e551e7 P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:44.567549 17471 raft_consensus.cc:357] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.567785 17471 raft_consensus.cc:383] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.567857 17471 raft_consensus.cc:738] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.568145 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet 163857e9dd0b446ba48262c908e551e7 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]) successfully deleted
I20250111 08:58:44.568100 17471 consensus_queue.cc:260] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
    @     0x7ff2398e04dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250111 08:58:44.570320 17475 tablet_bootstrap.cc:492] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.570457 17475 ts_tablet_manager.cc:1397] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.035s	user 0.000s	sys 0.005s
I20250111 08:58:44.571197 17475 raft_consensus.cc:357] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.571399 17475 raft_consensus.cc:383] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.571485 17475 raft_consensus.cc:738] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.573323 17322 tablet_service.cc:1514] Processing DeleteTablet for tablet 3586e89f11214f869d9922fe7d8a5bca with delete_type TABLET_DATA_DELETED (replaced by tablet b43238f36aa44f87b625287daef14f76) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.573693 17471 ts_tablet_manager.cc:1428] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
W20250111 08:58:44.573872 16723 catalog_manager.cc:4852] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): delete failed for tablet 3586e89f11214f869d9922fe7d8a5bca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3586e89f11214f869d9922fe7d8a5bca already in progress: creating tablet
    @     0x7ff2398e0359  std::_Function_handler<>::_M_invoke() at ??:0
I20250111 08:58:44.574589 17475 consensus_queue.cc:260] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
    @           0xa71ecd  std::function<>::operator()() at ??:?
I20250111 08:58:44.577186 17475 ts_tablet_manager.cc:1428] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20250111 08:58:44.577450 17475 tablet_bootstrap.cc:492] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2: Bootstrap starting.
I20250111 08:58:44.578598 17474 tablet_bootstrap.cc:492] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.578761 17474 ts_tablet_manager.cc:1397] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.040s	user 0.007s	sys 0.000s
I20250111 08:58:44.579353 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet 163857e9dd0b446ba48262c908e551e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3586e89f11214f869d9922fe7d8a5bca) from {username='slave'} at 127.0.0.1:40166
I20250111 08:58:44.579596 17322 tablet_service.cc:1514] Processing DeleteTablet for tablet 3586e89f11214f869d9922fe7d8a5bca with delete_type TABLET_DATA_DELETED (replaced by tablet b43238f36aa44f87b625287daef14f76) from {username='slave'} at 127.0.0.1:50330
I20250111 08:58:44.579799 17481 tablet_replica.cc:331] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: stopping tablet replica
I20250111 08:58:44.580147 17474 raft_consensus.cc:357] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.580323 17481 raft_consensus.cc:2238] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.580348 17474 raft_consensus.cc:383] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.580464 17474 raft_consensus.cc:738] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.580523 17481 raft_consensus.cc:2267] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.580732 17474 consensus_queue.cc:260] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.581439 17481 ts_tablet_manager.cc:1905] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.581538 17480 tablet_replica.cc:331] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:44.581751 17480 raft_consensus.cc:2238] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.581923 17480 raft_consensus.cc:2267] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.582631 17480 ts_tablet_manager.cc:1905] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:44.587078 17481 ts_tablet_manager.cc:1918] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.587190 17481 log.cc:1198] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-2/wal/wals/3586e89f11214f869d9922fe7d8a5bca
I20250111 08:58:44.587509 17481 ts_tablet_manager.cc:1939] T 3586e89f11214f869d9922fe7d8a5bca P 25e23fc83774472986af5c3f81f591fe: Deleting consensus metadata
I20250111 08:58:44.588035 16723 catalog_manager.cc:4872] TS 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627): tablet 3586e89f11214f869d9922fe7d8a5bca (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]) successfully deleted
    @     0x7ff2264950be  kudu::Thread::SuperviseThread() at ??:0
I20250111 08:58:44.590377 17475 tablet_bootstrap.cc:654] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.590757 17474 ts_tablet_manager.cc:1428] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.002s
I20250111 08:58:44.591003 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet 3586e89f11214f869d9922fe7d8a5bca with delete_type TABLET_DATA_DELETED (replaced by tablet b43238f36aa44f87b625287daef14f76) from {username='slave'} at 127.0.0.1:58534
I20250111 08:58:44.591534 17482 tablet_replica.cc:331] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:44.591710 17482 raft_consensus.cc:2238] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:44.592409 17482 raft_consensus.cc:2267] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:44.594035 17474 tablet_bootstrap.cc:492] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.595014 17482 ts_tablet_manager.cc:1905] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
    @     0x7ff2298786db  start_thread at ??:0
I20250111 08:58:44.595456 17480 ts_tablet_manager.cc:1918] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.595579 17480 log.cc:1198] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/163857e9dd0b446ba48262c908e551e7
I20250111 08:58:44.595909 17480 ts_tablet_manager.cc:1939] T 163857e9dd0b446ba48262c908e551e7 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
    @     0x7ff2225e871f  clone at ??:0
I20250111 08:58:44.597649 17475 tablet_bootstrap.cc:492] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2: No bootstrap required, opened a new log
I20250111 08:58:44.597793 17475 ts_tablet_manager.cc:1397] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.000s
I20250111 08:58:44.598254 17474 tablet_bootstrap.cc:654] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.598554 17475 raft_consensus.cc:357] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.598752 17475 raft_consensus.cc:383] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.598822 17475 raft_consensus.cc:738] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.599088 17475 consensus_queue.cc:260] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.599990 17471 tablet_bootstrap.cc:492] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.599593 17475 ts_tablet_manager.cc:1428] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20250111 08:58:44.607151 17471 tablet_bootstrap.cc:654] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.608975 17474 tablet_bootstrap.cc:492] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.609130 17474 ts_tablet_manager.cc:1397] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.015s	user 0.001s	sys 0.002s
I20250111 08:58:44.609939 17474 raft_consensus.cc:357] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.610139 17474 raft_consensus.cc:383] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.610215 17474 raft_consensus.cc:738] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.610488 17488 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:44.610621 17488 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.612573 17488 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:44.623586 17071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f683bacb4e4778b7ef52d75fdb0832" candidate_uuid: "d650c29a4cae41a490fd9aad87215be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13afd9f6a62a4efd831797a347bb31b2" is_pre_election: true
I20250111 08:58:44.624020 17071 raft_consensus.cc:2463] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d650c29a4cae41a490fd9aad87215be7 in term 0.
I20250111 08:58:44.624672 17143 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7; no voters: 
I20250111 08:58:44.625099 17488 raft_consensus.cc:2798] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 08:58:44.625200 17488 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:58:44.625274 17488 raft_consensus.cc:3054] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.628069 17488 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.649181 17071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f683bacb4e4778b7ef52d75fdb0832" candidate_uuid: "d650c29a4cae41a490fd9aad87215be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13afd9f6a62a4efd831797a347bb31b2"
I20250111 08:58:44.649398 17071 raft_consensus.cc:3054] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.654630 17488 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [CANDIDATE]: Term 1 election: Requested vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:44.655067 17071 raft_consensus.cc:2463] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d650c29a4cae41a490fd9aad87215be7 in term 1.
I20250111 08:58:44.655561 17143 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7; no voters: 
I20250111 08:58:44.657824 17488 raft_consensus.cc:2798] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 08:58:44.658912 17488 raft_consensus.cc:695] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 1 LEADER]: Becoming Leader. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Running, Role: LEADER
I20250111 08:58:44.659384 17488 consensus_queue.cc:237] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.669061 16873 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=39ed1d0a4a7d415b9dba140fb6bbcec2]
I20250111 08:58:44.669402 16873 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 6) took 2401 ms. Trace:
I20250111 08:58:44.669513 16873 rpcz_store.cc:276] 0111 08:58:42.267562 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.268271 (+   709us) service_pool.cc:225] Handling call
0111 08:58:42.705582 (+437311us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.705897 (+   315us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.722412 (+1016515us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.723251 (+   839us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.723255 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:43.723464 (+   209us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.669359 (+945895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.723167 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.723181 (+    14us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.723190 (+     9us) write_op.cc:454] Released schema lock
0111 08:58:43.723196 (+     6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1670827,"Hive Metastore.run_cpu_time_us":456,"Hive Metastore.run_wall_time_us":290582,"child_traces":[["op",{}]]}
I20250111 08:58:44.677506 17474 consensus_queue.cc:260] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.678225 17474 ts_tablet_manager.cc:1428] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.069s	user 0.001s	sys 0.001s
I20250111 08:58:44.678509 17474 tablet_bootstrap.cc:492] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7: Bootstrap starting.
I20250111 08:58:44.679096 17600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.189:35063 every 8 connection(s)
I20250111 08:58:44.679400 17453 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.189:35063
I20250111 08:58:44.680166 17474 tablet_bootstrap.cc:654] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.683003 17453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
I20250111 08:58:44.685652 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 17453
I20250111 08:58:44.695945 17482 ts_tablet_manager.cc:1918] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:44.696091 17482 log.cc:1198] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/3586e89f11214f869d9922fe7d8a5bca
I20250111 08:58:44.696462 17482 ts_tablet_manager.cc:1939] T 3586e89f11214f869d9922fe7d8a5bca P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:44.720219 17471 tablet_bootstrap.cc:492] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.720424 17471 ts_tablet_manager.cc:1397] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.121s	user 0.003s	sys 0.003s
I20250111 08:58:44.721202 17471 raft_consensus.cc:357] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.721410 17471 raft_consensus.cc:383] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.721482 17471 raft_consensus.cc:738] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.724164 17471 consensus_queue.cc:260] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.728176 17486 raft_consensus.cc:491] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:44.728348 17486 raft_consensus.cc:513] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.729142 17486 leader_election.cc:290] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:44.729595 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634d4c9c6e846ae8c655b59f8d001d8" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe" is_pre_election: true
I20250111 08:58:44.729727 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634d4c9c6e846ae8c655b59f8d001d8" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7" is_pre_election: true
I20250111 08:58:44.729797 17343 raft_consensus.cc:2463] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 0.
I20250111 08:58:44.729936 17209 raft_consensus.cc:2463] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 0.
I20250111 08:58:44.730217 17004 leader_election.cc:304] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, 25e23fc83774472986af5c3f81f591fe; no voters: 
I20250111 08:58:44.730664 17486 raft_consensus.cc:2798] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 08:58:44.730769 17486 raft_consensus.cc:491] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:58:44.730859 17486 raft_consensus.cc:3054] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.734025 17486 raft_consensus.cc:513] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.735194 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634d4c9c6e846ae8c655b59f8d001d8" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7"
I20250111 08:58:44.735375 17209 raft_consensus.cc:3054] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.735836 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6634d4c9c6e846ae8c655b59f8d001d8" candidate_uuid: "13afd9f6a62a4efd831797a347bb31b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe"
I20250111 08:58:44.736009 17343 raft_consensus.cc:3054] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.738104 17209 raft_consensus.cc:2463] T 6634d4c9c6e846ae8c655b59f8d001d8 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 1.
I20250111 08:58:44.738441 17006 leader_election.cc:304] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7; no voters: 
I20250111 08:58:44.738543 17343 raft_consensus.cc:2463] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13afd9f6a62a4efd831797a347bb31b2 in term 1.
I20250111 08:58:44.739573 17486 leader_election.cc:290] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [CANDIDATE]: Term 1 election: Requested vote from peers d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105), 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:44.740466 17486 raft_consensus.cc:2798] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 08:58:44.741112 17486 raft_consensus.cc:695] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 LEADER]: Becoming Leader. State: Replica: 13afd9f6a62a4efd831797a347bb31b2, State: Running, Role: LEADER
I20250111 08:58:44.741415 17486 consensus_queue.cc:237] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.748286 17256 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:44.748754 17471 ts_tablet_manager.cc:1428] T 6634d4c9c6e846ae8c655b59f8d001d8 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.028s	user 0.002s	sys 0.000s
I20250111 08:58:44.749063 17471 tablet_bootstrap.cc:492] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe: Bootstrap starting.
I20250111 08:58:44.754069 17601 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:58:44.756631 17471 tablet_bootstrap.cc:654] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe: Neither blocks nor log segments found. Creating new log.
I20250111 08:58:44.757844 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f683bacb4e4778b7ef52d75fdb0832" candidate_uuid: "d650c29a4cae41a490fd9aad87215be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe" is_pre_election: true
I20250111 08:58:44.758049 17343 raft_consensus.cc:2463] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d650c29a4cae41a490fd9aad87215be7 in term 0.
I20250111 08:58:44.758293 17474 tablet_bootstrap.cc:492] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7: No bootstrap required, opened a new log
I20250111 08:58:44.758456 17474 ts_tablet_manager.cc:1397] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7: Time spent bootstrapping tablet: real 0.080s	user 0.003s	sys 0.000s
I20250111 08:58:44.758471 17601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap starting.
I20250111 08:58:44.758452 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f683bacb4e4778b7ef52d75fdb0832" candidate_uuid: "d650c29a4cae41a490fd9aad87215be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e23fc83774472986af5c3f81f591fe"
I20250111 08:58:44.758620 17343 raft_consensus.cc:3054] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.759291 17474 raft_consensus.cc:357] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.759495 17474 raft_consensus.cc:383] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.759558 17474 raft_consensus.cc:738] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d650c29a4cae41a490fd9aad87215be7, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.759801 17474 consensus_queue.cc:260] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.760293 17474 ts_tablet_manager.cc:1428] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20250111 08:58:44.761514 17343 raft_consensus.cc:2463] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d650c29a4cae41a490fd9aad87215be7 in term 1.
I20250111 08:58:44.766280 17471 tablet_bootstrap.cc:492] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe: No bootstrap required, opened a new log
I20250111 08:58:44.766434 17471 ts_tablet_manager.cc:1397] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe: Time spent bootstrapping tablet: real 0.017s	user 0.003s	sys 0.000s
I20250111 08:58:44.767201 17471 raft_consensus.cc:357] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.767441 17471 raft_consensus.cc:383] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 08:58:44.767508 17471 raft_consensus.cc:738] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.767839 17601 log.cc:826] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:44.769735 17471 consensus_queue.cc:260] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.770951 17471 ts_tablet_manager.cc:1428] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20250111 08:58:44.778497 17118 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:44.781241 17391 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:44.790571 17601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 08:58:44.791217 17601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap complete.
I20250111 08:58:44.799068 17601 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:44.799800 17601 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Initialized, Role: FOLLOWER
I20250111 08:58:44.800468 17601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:44.804170 17613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:44.805835 17601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:58:44.804540 17613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:44.816999 16870 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=20cc955c3d054cc9b4199caad238843b]
I20250111 08:58:44.817348 16870 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 9) took 2549 ms. Trace:
I20250111 08:58:44.817474 16870 rpcz_store.cc:276] 0111 08:58:42.267862 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.269099 (+  1237us) service_pool.cc:225] Handling call
0111 08:58:42.706315 (+437216us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.706608 (+   293us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.796407 (+1089799us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.797108 (+   701us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.797112 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:43.797303 (+   191us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.817305 (+1020002us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.797040 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.797050 (+    10us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.797055 (+     5us) write_op.cc:454] Released schema lock
0111 08:58:43.797059 (+     4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1886983,"Hive Metastore.run_cpu_time_us":488,"Hive Metastore.run_wall_time_us":222160,"spinlock_wait_cycles":43392,"child_traces":[["op",{}]]}
I20250111 08:58:44.867851 17473 raft_consensus.cc:491] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:44.868079 17473 raft_consensus.cc:513] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.870079 17473 leader_election.cc:290] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:44.878504 17071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43238f36aa44f87b625287daef14f76" candidate_uuid: "25e23fc83774472986af5c3f81f591fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13afd9f6a62a4efd831797a347bb31b2" is_pre_election: true
I20250111 08:58:44.878806 17071 raft_consensus.cc:2463] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e23fc83774472986af5c3f81f591fe in term 0.
I20250111 08:58:44.879580 17279 leader_election.cc:304] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, 25e23fc83774472986af5c3f81f591fe; no voters: 
I20250111 08:58:44.880146 17473 raft_consensus.cc:2798] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 08:58:44.880272 17473 raft_consensus.cc:491] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:58:44.880379 17473 raft_consensus.cc:3054] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.883409 17473 raft_consensus.cc:513] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.884588 17071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43238f36aa44f87b625287daef14f76" candidate_uuid: "25e23fc83774472986af5c3f81f591fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13afd9f6a62a4efd831797a347bb31b2"
I20250111 08:58:44.884814 17071 raft_consensus.cc:3054] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.887663 17071 raft_consensus.cc:2463] T b43238f36aa44f87b625287daef14f76 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e23fc83774472986af5c3f81f591fe in term 1.
I20250111 08:58:44.888154 17279 leader_election.cc:304] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13afd9f6a62a4efd831797a347bb31b2, 25e23fc83774472986af5c3f81f591fe; no voters: 
I20250111 08:58:44.892012 17473 leader_election.cc:290] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 1 election: Requested vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:44.892722 17473 raft_consensus.cc:2798] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader election won for term 1
I20250111 08:58:44.893160 17473 raft_consensus.cc:695] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [term 1 LEADER]: Becoming Leader. State: Replica: 25e23fc83774472986af5c3f81f591fe, State: Running, Role: LEADER
I20250111 08:58:44.893824 17473 consensus_queue.cc:237] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:44.904780 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43238f36aa44f87b625287daef14f76" candidate_uuid: "25e23fc83774472986af5c3f81f591fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7" is_pre_election: true
I20250111 08:58:44.905035 17209 raft_consensus.cc:2463] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25e23fc83774472986af5c3f81f591fe in term 0.
I20250111 08:58:44.905668 17209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43238f36aa44f87b625287daef14f76" candidate_uuid: "25e23fc83774472986af5c3f81f591fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d650c29a4cae41a490fd9aad87215be7"
I20250111 08:58:44.905861 17209 raft_consensus.cc:3054] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7 [term 0 FOLLOWER]: Advancing to term 1
I20250111 08:58:44.906102 16869 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=b90143630fb14428b06c0f2828bd6d6a]
I20250111 08:58:44.906435 16869 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 10) took 2638 ms. Trace:
I20250111 08:58:44.906548 16869 rpcz_store.cc:276] 0111 08:58:42.268001 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.269045 (+  1044us) service_pool.cc:225] Handling call
0111 08:58:42.706963 (+437918us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.707256 (+   293us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.859334 (+1152078us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.860124 (+   790us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.860128 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:43.860318 (+   190us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.906393 (+1046075us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.860052 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.860064 (+    12us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.860070 (+     6us) write_op.cc:454] Released schema lock
0111 08:58:43.860074 (+     4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":2045544,"Hive Metastore.run_cpu_time_us":498,"Hive Metastore.run_wall_time_us":151970,"child_traces":[["op",{}]]}
I20250111 08:58:44.908299 17209 raft_consensus.cc:2463] T b43238f36aa44f87b625287daef14f76 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e23fc83774472986af5c3f81f591fe in term 1.
I20250111 08:58:44.911969 17633 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250111 08:58:44.946942 16877 catalog_manager.cc:2159] deleted orphaned HMS catalog entry for table  [id=1620935ff43440ec8c2fb76207fce81d]
I20250111 08:58:44.947345 16877 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50996 (request call id 2) took 2688 ms. Trace:
I20250111 08:58:44.947482 16877 rpcz_store.cc:276] 0111 08:58:42.258632 (+     0us) service_pool.cc:168] Inserting onto call queue
0111 08:58:42.258921 (+   289us) service_pool.cc:225] Handling call
0111 08:58:42.261796 (+  2875us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x616000051a54. stack: 00007fad7fba8fd6 00007fad7fba8d90 00007fad7eb8848d 000000000096d544 000000000096d50d 00007fad8e746e1d 00007fad8e746f13 00007fad8b44f211 00007fad92f269dd 00007fad930b1d03 00007fad8a1b4236 00007fad8a1b3f65 00007fad8691030e 00007fad8690f1ee 00007fad86915425 00007fad86915d4f
0111 08:58:42.707342 (+445546us) catalog_manager.cc:2076] Acquired catalog manager lock
0111 08:58:42.707617 (+   275us) catalog_manager.cc:2122] Created new table and tablet info
0111 08:58:43.910373 (+1202756us) catalog_manager.cc:2148] Created new table in HMS catalog
0111 08:58:43.911156 (+   783us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.911160 (+     4us) write_op.cc:454] Released schema lock
0111 08:58:43.911359 (+   199us) catalog_manager.cc:2156] Rolling back HMS table creation
0111 08:58:44.947298 (+1035939us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 08:58:43.911079 (+     0us) op_driver.cc:389] HandleFailure(Illegal state: replica 4acb4ec5faa3472f9cecb23f9b5d2e0f is not leader of this config: current role FOLLOWER)
0111 08:58:43.911092 (+    13us) write_op.cc:489] Releasing partition, row and schema locks
0111 08:58:43.911099 (+     7us) write_op.cc:454] Released schema lock
0111 08:58:43.911104 (+     5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":2146243,"Hive Metastore.run_cpu_time_us":466,"Hive Metastore.run_wall_time_us":91651,"spinlock_wait_cycles":5790080,"child_traces":[["op",{}]]}
W20250111 08:58:44.980685 17280 connection.cc:537] client connection to 127.16.20.190:46487 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:44.980705 17400 connection.cc:537] client connection to 127.16.20.190:46487 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:44.981163 17144 connection.cc:537] client connection to 127.16.20.190:46487 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:44.981246 17400 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.190:46487)
W20250111 08:58:44.982466 17400 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.190:46487)
W20250111 08:58:44.982514 17006 connection.cc:537] client connection to 127.16.20.190:46487 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:44.983095 17006 connection.cc:537] server connection from 127.0.0.1:40166 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:44.983101 17144 connection.cc:537] server connection from 127.0.0.1:58534 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:44.984349 17459 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 08:58:44.981508 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.20.190:46487 (local address 127.16.20.131:56211)
0111 08:58:44.981805 (+   297us) negotiation.cc:107] Waiting for socket to connect
0111 08:58:44.981825 (+    20us) client_negotiation.cc:174] Beginning negotiation
0111 08:58:44.981958 (+   133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 08:58:44.983540 (+  1582us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":165}
W20250111 08:58:44.984393 17463 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 08:58:44.981949 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.20.190:46487 (local address 127.16.20.130:59009)
0111 08:58:44.982183 (+   234us) negotiation.cc:107] Waiting for socket to connect
0111 08:58:44.982200 (+    17us) client_negotiation.cc:174] Beginning negotiation
0111 08:58:44.982297 (+    97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 08:58:44.983522 (+  1225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":115}
W20250111 08:58:44.984791 17638 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 08:58:44.914060 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.20.190:46487 (local address 127.0.0.1:34586)
0111 08:58:44.914879 (+   819us) negotiation.cc:107] Waiting for socket to connect
0111 08:58:44.914913 (+    34us) client_negotiation.cc:174] Beginning negotiation
0111 08:58:44.915055 (+   142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0111 08:58:44.983709 (+ 68654us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":395,"threads_started":1}
I20250111 08:58:44.985123 17453 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107), 
I20250111 08:58:44.990561 17453 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:45.047117 17640 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:58:45.047281 17640 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:45.047878 17640 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:45.049855 16865 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:45.054512 17547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" is_pre_election: true
I20250111 08:58:45.055032 17547 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 2.
I20250111 08:58:45.055441 16864 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f; no voters: c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:45.055719 17640 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250111 08:58:45.055832 17640 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Starting leader election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:58:45.055936 17640 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 2 FOLLOWER]: Advancing to term 3
I20250111 08:58:45.057750 17640 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:45.058255 17640 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:58:45.058722 17547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
I20250111 08:58:45.058915 17547 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 2 FOLLOWER]: Advancing to term 3
W20250111 08:58:45.059340 16865 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:45.062703 17547 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 3.
I20250111 08:58:45.063035 16864 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f; no voters: c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:45.063396 17640 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 3 FOLLOWER]: Leader election won for term 3
I20250111 08:58:45.063823 17640 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 3 LEADER]: Becoming Leader. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Running, Role: LEADER
I20250111 08:58:45.064141 17640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.36, Last appended by leader: 36, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:45.065200 17644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb4ec5faa3472f9cecb23f9b5d2e0f. Latest consensus state: current_term: 3 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:45.065536 17644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:45.066066 17645 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:58:45.067888 17645 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:58:45.068290 17645 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:58:45.068648 17645 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:58:45.070416 17645 tablet_loader.cc:96] loaded metadata for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:58:45.071029 17645 tablet_loader.cc:96] loaded metadata for tablet a2f683bacb4e4778b7ef52d75fdb0832 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:58:45.071621 17645 tablet_loader.cc:96] loaded metadata for tablet b43238f36aa44f87b625287daef14f76 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:58:45.072463 17645 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:58:45.073354 17645 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:45.073426 17645 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:58:45.074726 17645 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:58:45.075549 17645 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Loaded TSK: 0
I20250111 08:58:45.076122 17645 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:58:45.076946 17645 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:58:45.077778 17645 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:58:45.080924 16877 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:45.080989 16869 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:45.081423 16873 master_service.cc:946] ReplaceTablet: received request to replace tablet a2f683bacb4e4778b7ef52d75fdb0832 from {username='slave'} at 127.0.0.1:50996
I20250111 08:58:45.081554 16876 master_service.cc:946] ReplaceTablet: received request to replace tablet a2f683bacb4e4778b7ef52d75fdb0832 from {username='slave'} at 127.0.0.1:50996
I20250111 08:58:45.081806 16875 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:45.081935 16878 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:45.082233 16871 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:50996
I20250111 08:58:45.082302 16872 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:45.081321 16870 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50996:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:45.083505 16874 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:50996
W20250111 08:58:45.086855 16865 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f -> Peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Couldn't send request to peer c77238c1f3c9423caf1235b18b67ab2e. Status: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 08:58:45.088680 17547 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Refusing update from remote peer 4acb4ec5faa3472f9cecb23f9b5d2e0f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 38. (index mismatch)
I20250111 08:58:45.089447 16934 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:58:45.089743 16934 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:58:45.089807 16934 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
I20250111 08:58:45.091058 17644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 4, Time since last communication: 0.000s
    @     0x7fad7ce70dcd  google::LogMessage::Fail() at ??:0
    @     0x7fad7ce74b93  google::LogMessage::SendToLog() at ??:0
    @     0x7fad7ce707cc  google::LogMessage::Flush() at ??:0
    @     0x7fad7ce70a11  google::LogMessage::~LogMessage() at ??:0
    @     0x7fad930136de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250111 08:58:45.106403 17651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:45.106756 17651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:45.107110 17613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb4ec5faa3472f9cecb23f9b5d2e0f. Latest consensus state: current_term: 3 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:45.107352 17613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
    @     0x7fad930123c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250111 08:58:45.111765 16873 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Sending DeleteTablet for 3 replicas of tablet a2f683bacb4e4778b7ef52d75fdb0832
    @     0x7fad930194dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250111 08:58:45.114212 16873 catalog_manager.cc:6596] ReplaceTablet: tablet a2f683bacb4e4778b7ef52d75fdb0832 deleted and replaced by tablet 30dabd99c09f4a0099a111c8e8512f52
I20250111 08:58:45.114396 17613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:45.114585 17613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
    @     0x7fad93019359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
I20250111 08:58:45.122300 17051 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f683bacb4e4778b7ef52d75fdb0832 with delete_type TABLET_DATA_DELETED (replaced by tablet 30dabd99c09f4a0099a111c8e8512f52) from {username='slave'} at 127.0.0.1:40178
I20250111 08:58:45.122978 17189 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f683bacb4e4778b7ef52d75fdb0832 with delete_type TABLET_DATA_DELETED (replaced by tablet 30dabd99c09f4a0099a111c8e8512f52) from {username='slave'} at 127.0.0.1:58546
I20250111 08:58:45.123291 17482 tablet_replica.cc:331] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: stopping tablet replica
I20250111 08:58:45.123494 17482 raft_consensus.cc:2238] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 1 LEADER]: Raft consensus shutting down.
I20250111 08:58:45.123945 17482 pending_rounds.cc:62] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Trying to abort 1 pending ops.
I20250111 08:58:45.124117 17482 pending_rounds.cc:69] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113055947438108672 op_type: NO_OP noop_request { }
I20250111 08:58:45.124292 17482 raft_consensus.cc:2883] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 08:58:45.124433 17482 raft_consensus.cc:2267] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7 [term 1 FOLLOWER]: Raft consensus is shut down!
    @     0x7fad7fbce0be  kudu::Thread::SuperviseThread() at ??:0
I20250111 08:58:45.125595 17482 ts_tablet_manager.cc:1905] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Deleting tablet data with delete state TABLET_DATA_DELETED
    @     0x7fad82fb16db  start_thread at ??:0
    @     0x7fad7bd2171f  clone at ??:0
I20250111 08:58:45.132092 17482 ts_tablet_manager.cc:1918] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 08:58:45.132212 17482 log.cc:1198] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-1/wal/wals/a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:58:45.132658 17482 ts_tablet_manager.cc:1939] T a2f683bacb4e4778b7ef52d75fdb0832 P d650c29a4cae41a490fd9aad87215be7: Deleting consensus metadata
I20250111 08:58:45.134660 17659 tablet_replica.cc:331] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: stopping tablet replica
I20250111 08:58:45.134805 17659 raft_consensus.cc:2238] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 08:58:45.134974 17659 raft_consensus.cc:2267] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 08:58:45.135373 17659 ts_tablet_manager.cc:1905] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 08:58:45.140275 17659 ts_tablet_manager.cc:1918] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 08:58:45.146373 17659 log.cc:1198] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: Deleting WAL directory at /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/ts-0/wal/wals/a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:58:45.146943 17659 ts_tablet_manager.cc:1939] T a2f683bacb4e4778b7ef52d75fdb0832 P 13afd9f6a62a4efd831797a347bb31b2: Deleting consensus metadata
I20250111 08:58:45.151741 17648 mvcc.cc:204] Tried to move back new op lower bound from 7113055949149499392 to 7113055937692499968. Current Snapshot: MvccSnapshot[applied={T|T < 7113055949149499392}]
I20250111 08:58:45.271762 17530 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:34069; Asking this server to re-register.
I20250111 08:58:45.272579 17256 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:45.274562 17530 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:45.282243 17530 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:36067; Asking this server to re-register.
I20250111 08:58:45.282853 17118 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:45.284106 17530 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
W20250111 08:58:45.295576 17142 connection.cc:537] client connection to 127.16.20.188:41785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:45.295794 17589 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 08:58:45.116317 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.16.20.131:33627)
0111 08:58:45.116494 (+   177us) server_negotiation.cc:206] Beginning negotiation
0111 08:58:45.116500 (+     6us) server_negotiation.cc:399] Waiting for connection header
0111 08:58:45.121145 (+  4645us) server_negotiation.cc:407] Connection header received
0111 08:58:45.123747 (+  2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0111 08:58:45.123754 (+     7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0111 08:58:45.123837 (+    83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0111 08:58:45.123907 (+    70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0111 08:58:45.295557 (+171650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50338: BlockingRecv error: recv got EOF from 127.0.0.1:50338 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20250111 08:58:45.296185 17003 connection.cc:537] server connection from 127.0.0.1:40178 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:45.296950 17004 connection.cc:537] client connection to 127.16.20.188:41785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 08:58:45.412684 17530 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:52067; Asking this server to re-register.
I20250111 08:58:45.413246 17391 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:45.414268 17530 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:45.615173 17671 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:45.615368 17671 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:45.617067 17671 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:58:45.617874 17516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:58:45.618958 17516 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:58:45.619135 17511 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:45.619256 17511 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:45.619570 17671 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250111 08:58:45.760406 17675 consensus_queue.cc:1035] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 08:58:45.768571 17679 consensus_queue.cc:1035] T 6634d4c9c6e846ae8c655b59f8d001d8 P 13afd9f6a62a4efd831797a347bb31b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 08:58:45.792749 17676 consensus_queue.cc:1035] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 08:58:45.799387 17690 consensus_queue.cc:1035] T b43238f36aa44f87b625287daef14f76 P 25e23fc83774472986af5c3f81f591fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 08:58:45.916316 17633 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:45.916464 17633 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: loading cluster ID for follower catalog manager: success
I20250111 08:58:45.919216 17633 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: acquiring CA information for follower catalog manager: success
I20250111 08:58:45.921162 17633 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:58:46.159093 17693 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:46.159307 17693 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:46.162169 17693 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:58:46.163367 17516 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:58:46.163753 17511 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:46.163903 17511 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:46.164163 17693 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250111 08:58:47.235332 17715 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:47.235533 17715 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:47.236157 17715 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:58:47.238075 17511 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
W20250111 08:58:47.238236 17516 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:47.238380 17516 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:47.238647 17715 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250111 08:58:47.937697 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--webserver_interface=127.16.20.188
--webserver_port=41249
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 08:58:48.249044 17727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:48.249320 17727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:48.249444 17727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:48.257563 17727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:48.257661 17727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:48.257727 17727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:48.257802 17727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:48.270450 17727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.20.188
--webserver_port=41249
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:48.272392 17727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:48.274413 17727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:48.285212 17734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:48.285794 17737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:48.286365 17735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:48.287308 17727 server_base.cc:1034] running on GCE node
I20250111 08:58:48.288014 17727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:48.289292 17727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:48.290524 17727 hybrid_clock.cc:648] HybridClock initialized: now 1736585928290456 us; error 34 us; skew 500 ppm
I20250111 08:58:48.290834 17727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:48.293462 17727 webserver.cc:458] Webserver started at http://127.16.20.188:41249/ using document root <none> and password file <none>
I20250111 08:58:48.294087 17727 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:48.294193 17727 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:48.299003 17727 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20250111 08:58:48.301519 17744 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:48.302696 17727 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20250111 08:58:48.302822 17727 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
format_stamp: "Formatted at 2025-01-11 08:58:40 on dist-test-slave-s924"
I20250111 08:58:48.303586 17727 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:48.321125 17727 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:48.321861 17727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:48.322085 17727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:48.337572 17727 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.188:41785
I20250111 08:58:48.337604 17797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.188:41785 every 8 connection(s)
I20250111 08:58:48.338910 17727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
I20250111 08:58:48.344127 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 17727
I20250111 08:58:48.346637 17798 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:58:48.351394 17798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap starting.
I20250111 08:58:48.357765 17116 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:58:48.363826 17798 log.cc:826] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:48.392475 17798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=0 applied=38 ignored=0} inserts{seen=25 ignored=0} mutations{seen=26 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 08:58:48.392984 17798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap complete.
I20250111 08:58:48.399060 17798 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:48.401706 17798 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Initialized, Role: FOLLOWER
I20250111 08:58:48.402314 17798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 3.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:48.402993 17805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:48.403208 17805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:48.403918 17798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:58:48.420866 17820 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:48.421031 17820 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: loading cluster ID for follower catalog manager: success
I20250111 08:58:48.425135 17820 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: acquiring CA information for follower catalog manager: success
I20250111 08:58:48.427901 17820 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:58:48.430940 17727 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:58:48.437251 17727 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:48.637681 17389 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:58:48.638602 17740 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:48.638769 17740 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:48.639441 17740 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:48.640403 17763 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:54281; Asking this server to re-register.
W20250111 08:58:48.640700 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:58:48.640946 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:58:48.641053 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 08:58:48.641322 17740 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:58:48.641743 17389 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:48.643226 17763 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:48.777328 17824 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:48.777541 17824 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:48.778204 17824 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:58:48.780167 17516 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:48.784461 17773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:48.784948 17773 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 3.
I20250111 08:58:48.785557 17511 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f; no voters: c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:48.785823 17824 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250111 08:58:48.785933 17824 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4acb4ec5faa3472f9cecb23f9b5d2e0f)
I20250111 08:58:48.786011 17824 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 3 FOLLOWER]: Advancing to term 4
I20250111 08:58:48.787992 17824 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:48.788609 17824 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:48.789072 17773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
I20250111 08:58:48.789255 17773 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 3 FOLLOWER]: Advancing to term 4
W20250111 08:58:48.789654 17516 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:48.792754 17773 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 4.
I20250111 08:58:48.793092 17511 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f; no voters: c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:48.793360 17824 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Leader election won for term 4
I20250111 08:58:48.793711 17824 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 LEADER]: Becoming Leader. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Running, Role: LEADER
I20250111 08:58:48.794098 17824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 39, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:48.795615 17828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77aa4f8aaf704f1898426e17c6fbe28f. Latest consensus state: current_term: 4 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:48.795964 17828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:48.796605 17830 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:58:48.798909 17830 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:58:48.799350 17830 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:58:48.799681 17830 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:58:48.801482 17830 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:58:48.801936 17830 tablet_loader.cc:96] loaded metadata for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:58:48.802577 17830 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:58:48.803988 17830 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:58:48.804966 17830 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:48.805054 17830 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:58:48.806339 17830 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:58:48.807658 17830 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Loaded TSK: 0
I20250111 08:58:48.808599 17830 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:58:48.809748 17830 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:58:48.810673 17830 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:58:48.812842 17254 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:58:48.813968 17763 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:40527; Asking this server to re-register.
I20250111 08:58:48.814432 17254 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:48.815557 17763 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:48.843421 17530 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36008:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:58:48.843411 17529 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36008:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:58:48.844116 17527 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.844751 17528 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.845237 17521 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.845518 17526 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.845644 17522 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.846124 17525 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.846391 17524 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:36008
I20250111 08:58:48.846421 17523 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:36008
I20250111 08:58:48.847455 17523 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.848563 17524 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36008:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:48.851662 17631 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:58:48.852026 17631 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:58:48.852105 17631 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
    @     0x7f87a4d4cdcd  google::LogMessage::Fail() at ??:0
    @     0x7f87a4d50b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f87a4d4c7cc  google::LogMessage::Flush() at ??:0
    @     0x7f87a4d4ca11  google::LogMessage::~LogMessage() at ??:0
I20250111 08:58:48.862174 17763 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:33429; Asking this server to re-register.
I20250111 08:58:48.862887 17116 heartbeater.cc:463] Registering TS with master...
    @     0x7f87baeef6de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250111 08:58:48.864198 17763 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
    @     0x7f87baeee3c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
    @     0x7f87baef54dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
    @     0x7f87baef5359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
    @     0x7f87a7aaa0be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f87aae8d6db  start_thread at ??:0
    @     0x7f87a3bfd71f  clone at ??:0
W20250111 08:58:49.034348 17399 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.189:35063)
W20250111 08:58:49.034723 17399 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.189:35063)
W20250111 08:58:49.035173 17399 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.189:35063)
I20250111 08:58:49.345741 17847 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:49.345950 17847 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:49.347543 17847 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:49.348779 17749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:49.348912 17118 heartbeater.cc:502] Master 127.16.20.189:35063 was elected leader, sending a full tablet report...
W20250111 08:58:49.350188 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:58:49.350422 17750 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:49.350546 17750 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:49.350884 17847 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 08:58:49.855345 17852 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:49.855525 17852 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:49.856163 17852 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:49.857729 17750 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:58:49.857889 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:49.858060 17749 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:49.858287 17852 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 08:58:50.907497 17858 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:50.907696 17858 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:50.908231 17858 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:50.910519 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:58:50.910681 17750 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:50.910815 17750 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:50.911036 17858 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250111 08:58:51.433367 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--webserver_interface=127.16.20.189
--webserver_port=43097
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 08:58:51.741807 17864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:51.742082 17864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:51.742168 17864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:51.749436 17864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:51.749526 17864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:51.749574 17864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:51.749629 17864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:51.758351 17864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.20.189
--webserver_port=43097
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:51.759721 17864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:51.761303 17864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:51.771281 17870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:51.772131 17871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:51.772864 17873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:51.773506 17864 server_base.cc:1034] running on GCE node
I20250111 08:58:51.774173 17864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:51.775375 17864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:51.776579 17864 hybrid_clock.cc:648] HybridClock initialized: now 1736585931776522 us; error 31 us; skew 500 ppm
I20250111 08:58:51.776870 17864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:51.778908 17864 webserver.cc:458] Webserver started at http://127.16.20.189:43097/ using document root <none> and password file <none>
I20250111 08:58:51.779572 17864 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:51.779683 17864 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:51.784451 17864 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20250111 08:58:51.787094 17879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:51.788502 17864 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20250111 08:58:51.788648 17864 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:51.789397 17864 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:51.817813 17864 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:51.818539 17864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:51.818764 17864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:51.834250 17864 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.189:35063
I20250111 08:58:51.834290 17930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.189:35063 every 8 connection(s)
I20250111 08:58:51.835655 17864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
I20250111 08:58:51.839661 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 17864
I20250111 08:58:51.842684 17931 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:58:51.845813 17931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap starting.
I20250111 08:58:51.856678 17931 log.cc:826] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:51.859362 17118 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:51.859553 17118 heartbeater.cc:502] Master 127.16.20.189:35063 was elected leader, sending a full tablet report...
I20250111 08:58:51.885241 17931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=39 ignored=0} inserts{seen=26 ignored=0} mutations{seen=27 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 08:58:51.885790 17931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap complete.
I20250111 08:58:51.891903 17931 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:51.892840 17931 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Initialized, Role: FOLLOWER
I20250111 08:58:51.893383 17931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 4.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:51.894071 17935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:51.894680 17935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:51.895018 17931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:58:51.906689 17949 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:51.906831 17949 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: loading cluster ID for follower catalog manager: success
I20250111 08:58:51.910918 17949 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: acquiring CA information for follower catalog manager: success
I20250111 08:58:51.913605 17949 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:58:51.916152 17864 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:58:51.921125 17864 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:52.143324 17391 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:52.145501 17896 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:54701; Asking this server to re-register.
I20250111 08:58:52.146077 17391 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:52.147706 17896 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:52.319748 17256 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:52.320843 17896 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:41619; Asking this server to re-register.
I20250111 08:58:52.321331 17256 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:52.322374 17896 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:52.350450 17935 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:52.350646 17935 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:52.352309 17935 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:52.353613 17955 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250111 08:58:52.353561 17883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:52.353787 17955 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:52.354513 17955 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 08:58:52.355372 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:58:52.355538 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:58:52.355667 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
W20250111 08:58:52.355777 17883 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:52.355939 17955 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:58:52.358779 17773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 5 candidate_status { last_received { term: 4 index: 40 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:52.359002 17773 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 4.
I20250111 08:58:52.359547 17881 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f; no voters: c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:52.359894 17935 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250111 08:58:52.360028 17935 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:58:52.360113 17935 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 4 FOLLOWER]: Advancing to term 5
I20250111 08:58:52.362437 17118 heartbeater.cc:502] Master 127.16.20.189:35063 was elected leader, sending a full tablet report...
I20250111 08:58:52.364243 17935 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:52.364748 17896 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:57265; Asking this server to re-register.
I20250111 08:58:52.364857 17935 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 5 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:52.365314 17773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 5 candidate_status { last_received { term: 4 index: 40 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
I20250111 08:58:52.365420 17118 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:52.365525 17773 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 4 FOLLOWER]: Advancing to term 5
W20250111 08:58:52.366178 17883 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:52.366704 17896 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:58:52.367563 17773 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 5.
I20250111 08:58:52.367854 17881 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f; no voters: c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:52.368094 17935 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Leader election won for term 5
I20250111 08:58:52.368496 17935 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 LEADER]: Becoming Leader. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Running, Role: LEADER
I20250111 08:58:52.368865 17935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.40, Last appended by leader: 40, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:52.371171 17957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77aa4f8aaf704f1898426e17c6fbe28f. Latest consensus state: current_term: 5 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:52.371420 17957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:52.371927 17959 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:58:52.374284 17959 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:58:52.374702 17959 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:58:52.375062 17959 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:58:52.377046 17959 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:58:52.377449 17959 tablet_loader.cc:96] loaded metadata for tablet 6634d4c9c6e846ae8c655b59f8d001d8 (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:58:52.377943 17959 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:58:52.379222 17959 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:58:52.380095 17959 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:52.380162 17959 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:58:52.381361 17959 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:58:52.382186 17959 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Loaded TSK: 0
I20250111 08:58:52.382925 17959 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:58:52.383750 17959 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:58:52.384840 17959 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:58:52.425205 17896 master_service.cc:946] ReplaceTablet: received request to replace tablet 6634d4c9c6e846ae8c655b59f8d001d8 from {username='slave'} at 127.0.0.1:41070
I20250111 08:58:52.425330 17895 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:41070
I20250111 08:58:52.426390 17894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:52.426757 17895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:52.427495 17892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:52.427526 17893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:52.428761 17890 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41070:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:58:52.429261 17889 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41070:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:58:52.429824 17888 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:52.430478 17887 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:52.428189 17891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41070:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:52.434073 17948 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:58:52.434507 17948 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:58:52.434616 17948 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
I20250111 08:58:52.435307 17773 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Refusing update from remote peer 77aa4f8aaf704f1898426e17c6fbe28f: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
W20250111 08:58:52.436007 17883 consensus_peers.cc:487] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f -> Peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Couldn't send request to peer c77238c1f3c9423caf1235b18b67ab2e. Status: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 08:58:52.436148 17957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 38, Time since last communication: 0.000s
    @     0x7f63c7e8ddcd  google::LogMessage::Fail() at ??:0
    @     0x7f63c7e91b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f63c7e8d7cc  google::LogMessage::Flush() at ??:0
    @     0x7f63c7e8da11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f63de0306de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
    @     0x7f63de02f3c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250111 08:58:52.455935 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77aa4f8aaf704f1898426e17c6fbe28f. Latest consensus state: current_term: 5 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:52.456171 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
    @     0x7f63de0364dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250111 08:58:52.457876 17961 mvcc.cc:204] Tried to move back new op lower bound from 7113055979246825472 to 7113055978986049536. Current Snapshot: MvccSnapshot[applied={T|T < 7113055979246825472}]
    @     0x7f63de036359  std::_Function_handler<>::_M_invoke() at ??:0
I20250111 08:58:52.460954 17896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Sending DeleteTablet for 3 replicas of tablet 6634d4c9c6e846ae8c655b59f8d001d8
    @           0xa71ecd  std::function<>::operator()() at ??:?
I20250111 08:58:52.462836 17896 catalog_manager.cc:6596] ReplaceTablet: tablet 6634d4c9c6e846ae8c655b59f8d001d8 deleted and replaced by tablet c0fa97b400b14819b2a6e69865df8b8c
    @     0x7f63cabeb0be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f63cdfce6db  start_thread at ??:0
    @     0x7f63c6d3e71f  clone at ??:0
I20250111 08:58:52.478147 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:52.478390 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:52.478641 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:52.478885 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
W20250111 08:58:52.632671 17977 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 08:58:52.466095 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.16.20.131:33627)
0111 08:58:52.476765 (+ 10670us) server_negotiation.cc:206] Beginning negotiation
0111 08:58:52.476771 (+     6us) server_negotiation.cc:399] Waiting for connection header
0111 08:58:52.632303 (+155532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52320: BlockingRecv error: recv got EOF from 127.0.0.1:52320 (error 108)
Metrics: {"server-negotiator.queue_time_us":10558,"thread_start_us":10427,"threads_started":1}
W20250111 08:58:52.632735 17976 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 08:58:52.465710 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.16.20.130:35105)
0111 08:58:52.477085 (+ 11375us) server_negotiation.cc:206] Beginning negotiation
0111 08:58:52.477090 (+     5us) server_negotiation.cc:399] Waiting for connection header
0111 08:58:52.477106 (+    16us) server_negotiation.cc:407] Connection header received
0111 08:58:52.477163 (+    57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0111 08:58:52.477167 (+     4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0111 08:58:52.477248 (+    81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0111 08:58:52.477332 (+    84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0111 08:58:52.632336 (+155004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34848: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":11277,"thread_start_us":11173,"threads_started":1}
W20250111 08:58:52.633386 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0111 08:58:52.465437 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.16.20.129:37699)
0111 08:58:52.477015 (+ 11578us) server_negotiation.cc:206] Beginning negotiation
0111 08:58:52.477021 (+     6us) server_negotiation.cc:399] Waiting for connection header
0111 08:58:52.477039 (+    18us) server_negotiation.cc:407] Connection header received
0111 08:58:52.477106 (+    67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0111 08:58:52.477110 (+     4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0111 08:58:52.477185 (+    75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0111 08:58:52.477283 (+    98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0111 08:58:52.632315 (+155032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59518: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":11468,"thread_start_us":11345,"threads_started":1}
W20250111 08:58:52.633355 17749 connection.cc:537] server connection from 127.0.0.1:45724 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:52.634104 17399 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.189:35063)
W20250111 08:58:52.634474 17399 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.189:35063)
W20250111 08:58:52.634943 17399 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.189:35063)
I20250111 08:58:52.958912 17973 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:58:52.959084 17973 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:52.959645 17973 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:52.961380 17750 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:58:52.961591 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:52.961701 17749 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:52.961923 17973 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 08:58:53.438189 17727 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:58:53.438345 17727 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:53.668551 17992 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:58:53.668718 17992 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:53.669340 17992 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:53.671002 17750 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:58:53.671190 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:53.671286 17749 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:53.671497 17992 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 08:58:54.189000 17995 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:58:54.189157 17995 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:54.189739 17995 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:54.191779 17750 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:58:54.191998 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:54.192101 17749 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:54.192324 17995 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 08:58:55.329751 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.190:46487
--webserver_interface=127.16.20.190
--webserver_port=38281
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 08:58:55.632869 18002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:55.633149 18002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:55.633267 18002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:55.640889 18002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:55.641017 18002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:55.641090 18002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:55.641166 18002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:55.651147 18002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.190:46487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.20.190
--webserver_port=38281
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:55.652663 18002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:55.654361 18002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:55.664281 18007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:55.665357 18010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:55.666055 18008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:55.666680 18002 server_base.cc:1034] running on GCE node
I20250111 08:58:55.667439 18002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:55.668658 18002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:55.669853 18002 hybrid_clock.cc:648] HybridClock initialized: now 1736585935669798 us; error 29 us; skew 500 ppm
I20250111 08:58:55.670145 18002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:55.672142 18002 webserver.cc:458] Webserver started at http://127.16.20.190:38281/ using document root <none> and password file <none>
I20250111 08:58:55.672829 18002 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:55.672955 18002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:55.677620 18002 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.003s
I20250111 08:58:55.680039 18016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:55.681193 18002 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20250111 08:58:55.681324 18002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
uuid: "c77238c1f3c9423caf1235b18b67ab2e"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:55.682085 18002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:55.696655 18021 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:58:55.696823 18021 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:55.697517 18021 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:55.698498 17749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20250111 08:58:55.699873 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:58:55.700062 17750 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:58:55.700196 17750 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:58:55.700481 18021 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 08:58:55.702039 18002 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:55.702728 18002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:55.702965 18002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:55.717972 18002 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.190:46487
I20250111 08:58:55.717990 18071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.190:46487 every 8 connection(s)
I20250111 08:58:55.719285 18002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-0/data/info.pb
I20250111 08:58:55.723968 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 18002
I20250111 08:58:55.725322 18072 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:58:55.738169 18072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Bootstrap starting.
I20250111 08:58:55.752885 18072 log.cc:826] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:55.780082 18072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=24 ignored=0} mutations{seen=25 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 08:58:55.780568 18072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Bootstrap complete.
I20250111 08:58:55.786484 18072 raft_consensus.cc:357] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:55.787263 18072 raft_consensus.cc:738] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77238c1f3c9423caf1235b18b67ab2e, State: Initialized, Role: FOLLOWER
I20250111 08:58:55.787824 18072 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 2.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:55.788585 18077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:55.788836 18077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:55.789407 18072 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:58:55.799664 18091 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:55.799819 18091 catalog_manager.cc:1553] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: loading cluster ID for follower catalog manager: success
I20250111 08:58:55.803898 18091 catalog_manager.cc:1575] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: acquiring CA information for follower catalog manager: success
I20250111 08:58:55.806833 18091 catalog_manager.cc:1603] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:58:55.817102 18002 master_runner.cc:186] Error getting master registration for 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111), 
I20250111 08:58:55.822621 18002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:55.826634 17255 heartbeater.cc:346] Connected to a master server at 127.16.20.190:46487
I20250111 08:58:55.829324 18037 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:56905; Asking this server to re-register.
I20250111 08:58:55.829896 17255 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:55.831231 18037 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:55.861913 17117 heartbeater.cc:346] Connected to a master server at 127.16.20.190:46487
I20250111 08:58:55.863487 18037 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:56839; Asking this server to re-register.
I20250111 08:58:55.864051 17117 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:55.865231 18037 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:58:56.156534 17390 heartbeater.cc:346] Connected to a master server at 127.16.20.190:46487
I20250111 08:58:56.157847 18037 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:59823; Asking this server to re-register.
I20250111 08:58:56.158360 17390 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:56.159394 18037 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:56.313625 18096 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:56.313890 18096 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:56.315636 18096 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:58:56.316313 18019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:58:56.317589 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:56.321031 17773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:56.321274 17773 raft_consensus.cc:2363] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77238c1f3c9423caf1235b18b67ab2e for earlier term 3. Current term is 5.
I20250111 08:58:56.321825 18018 leader_election.cc:400] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77238c1f3c9423caf1235b18b67ab2e for earlier term 3. Current term is 5.
I20250111 08:58:56.321893 18018 leader_election.cc:403] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250111 08:58:56.322140 18096 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 2 FOLLOWER]: Advancing to term 5
I20250111 08:58:56.325690 18096 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77238c1f3c9423caf1235b18b67ab2e for earlier term 3. Current term is 5.
I20250111 08:58:56.966236 18107 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:56.966420 18107 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:56.967087 18107 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:56.967592 17773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:56.967891 17773 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77238c1f3c9423caf1235b18b67ab2e for term 6 because replica has last-logged OpId of term: 5 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 36.
W20250111 08:58:56.968887 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:56.969007 18019 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:58:56.969321 18107 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 08:58:57.448477 18114 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:57.448657 18114 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:58:57.449308 18114 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 08:58:57.450045 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:58:57.450268 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:58:57.450361 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 08:58:57.450591 18114 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:58:57.730312 18117 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:57.730489 18117 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:57.731117 18117 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:57.731631 17773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 6 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:58:57.731938 17773 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77238c1f3c9423caf1235b18b67ab2e for term 6 because replica has last-logged OpId of term: 5 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 36.
W20250111 08:58:57.732841 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:57.732946 18019 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:58:57.733168 18117 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 08:58:57.760201 18120 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:58:57.760385 18120 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:57.760914 18120 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:58:57.762898 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:57.767122 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:58:57.767494 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 5.
I20250111 08:58:57.768033 17750 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:58:57.768318 18120 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250111 08:58:57.768461 18120 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:58:57.768548 18120 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 5 FOLLOWER]: Advancing to term 6
I20250111 08:58:57.770421 18120 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:57.770937 18120 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:58:57.771289 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e"
I20250111 08:58:57.771462 18047 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 5 FOLLOWER]: Advancing to term 6
W20250111 08:58:57.772094 17749 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:58:57.773629 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 6.
I20250111 08:58:57.773959 17750 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:58:57.774210 18120 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 FOLLOWER]: Leader election won for term 6
I20250111 08:58:57.774564 18120 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 LEADER]: Becoming Leader. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Running, Role: LEADER
I20250111 08:58:57.774969 18120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:57.776427 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb4ec5faa3472f9cecb23f9b5d2e0f. Latest consensus state: current_term: 6 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:57.776734 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: LEADER
I20250111 08:58:57.777287 18127 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:58:57.779644 18127 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:58:57.780083 18127 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:58:57.780447 18127 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:58:57.782167 18127 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:58:57.783061 18127 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:58:57.783900 18127 tablet_loader.cc:96] loaded metadata for tablet c0fa97b400b14819b2a6e69865df8b8c (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:58:57.784826 18127 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:58:57.785703 18127 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:57.785776 18127 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:58:57.786948 18127 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:58:57.787824 18127 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Loaded TSK: 0
I20250111 08:58:57.788681 18127 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:58:57.789530 18127 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:58:57.790414 18127 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:58:57.854864 17762 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:51748
I20250111 08:58:57.854938 17763 master_service.cc:946] ReplaceTablet: received request to replace tablet 6634d4c9c6e846ae8c655b59f8d001d8 from {username='slave'} at 127.0.0.1:51748
I20250111 08:58:57.855993 17759 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.856163 17760 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.856324 17755 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.856905 17762 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.857216 17761 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.857815 17758 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.857971 17763 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.858567 17757 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51748:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:58:57.858800 17756 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51748:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:58:57.858932 17754 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51748:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:58:57.863391 17819 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:58:57.863813 17819 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:58:57.863912 17819 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
    @     0x7fad69ebddcd  google::LogMessage::Fail() at ??:0
    @     0x7fad69ec1b93  google::LogMessage::SendToLog() at ??:0
    @     0x7fad69ebd7cc  google::LogMessage::Flush() at ??:0
    @     0x7fad69ebda11  google::LogMessage::~LogMessage() at ??:0
    @     0x7fad800606de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
    @     0x7fad8005f3c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
    @     0x7fad800664dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
    @     0x7fad80066359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
    @     0x7fad6cc1b0be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fad6fffe6db  start_thread at ??:0
    @     0x7fad68d6e71f  clone at ??:0
W20250111 08:58:58.044530 17004 connection.cc:537] client connection to 127.16.20.188:41785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:58:58.044768 17398 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.188:41785)
W20250111 08:58:58.045135 17398 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.188:41785)
W20250111 08:58:58.045614 17398 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.188:41785)
I20250111 08:58:58.343942 17254 heartbeater.cc:502] Master 127.16.20.188:41785 was elected leader, sending a full tablet report...
I20250111 08:58:58.940721 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--webserver_interface=127.16.20.189
--webserver_port=43097
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 08:58:59.244091 18147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:58:59.244341 18147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:58:59.244462 18147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:58:59.251703 18147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:58:59.251787 18147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:58:59.251837 18147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:58:59.251892 18147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:58:59.261365 18147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.20.189
--webserver_port=43097
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:58:59.262724 18147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:58:59.264315 18147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:58:59.274511 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:59.275100 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:58:59.276014 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:58:59.276854 18147 server_base.cc:1034] running on GCE node
I20250111 08:58:59.277498 18147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:58:59.278662 18147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:58:59.279860 18147 hybrid_clock.cc:648] HybridClock initialized: now 1736585939279805 us; error 30 us; skew 500 ppm
I20250111 08:58:59.280151 18147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:58:59.282157 18147 webserver.cc:458] Webserver started at http://127.16.20.189:43097/ using document root <none> and password file <none>
I20250111 08:58:59.282805 18147 fs_manager.cc:362] Metadata directory not provided
I20250111 08:58:59.282907 18147 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:58:59.287664 18147 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.001s
I20250111 08:58:59.290171 18161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:58:59.291327 18147 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20250111 08:58:59.291458 18147 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:58:59.292244 18147 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:58:59.307116 18147 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:58:59.307821 18147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:58:59.308035 18147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:58:59.322922 18147 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.189:35063
I20250111 08:58:59.322942 18212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.189:35063 every 8 connection(s)
I20250111 08:58:59.324208 18147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
I20250111 08:58:59.329728 18213 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:58:59.332513 18213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap starting.
I20250111 08:58:59.334537 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 18147
I20250111 08:58:59.347448 18213 log.cc:826] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:58:59.348320 17256 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:59.378537 18213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=27 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 08:58:59.378999 18213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap complete.
I20250111 08:58:59.386030 17118 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:59.386698 18213 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:59.387491 18213 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Initialized, Role: FOLLOWER
I20250111 08:58:59.388074 18213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:59.388854 18219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:58:59.389118 18219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:58:59.389909 18213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:58:59.400269 18233 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:58:59.400439 18233 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: loading cluster ID for follower catalog manager: success
I20250111 08:58:59.404593 18233 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: acquiring CA information for follower catalog manager: success
I20250111 08:58:59.407191 18233 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:58:59.428264 18147 master_runner.cc:186] Error getting master registration for 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111), 
I20250111 08:58:59.435390 18147 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:58:59.669265 17391 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:58:59.671900 18177 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:44427; Asking this server to re-register.
I20250111 08:58:59.672546 17391 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:59.674309 18177 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:58:59.720746 18239 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:58:59.721006 18239 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:58:59.721843 18239 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:58:59.737697 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 7 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" is_pre_election: true
I20250111 08:58:59.738552 18188 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77238c1f3c9423caf1235b18b67ab2e for term 7 because replica has last-logged OpId of term: 5 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 36.
W20250111 08:58:59.740485 18018 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:58:59.740607 18018 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:58:59.740867 18239 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250111 08:58:59.851011 18177 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:57533; Asking this server to re-register.
I20250111 08:58:59.851486 17256 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:59.852537 18177 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:58:59.888113 18177 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:40457; Asking this server to re-register.
I20250111 08:58:59.888607 17118 heartbeater.cc:463] Registering TS with master...
I20250111 08:58:59.889596 18177 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:59:00.001665 18244 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:00.001930 18244 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:00.003640 18244 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:00.004106 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:59:00.004489 18047 raft_consensus.cc:2388] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77aa4f8aaf704f1898426e17c6fbe28f in current term 6: Already voted for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in this term.
W20250111 08:59:00.004566 18163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
W20250111 08:59:00.005860 18163 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:00.006003 18163 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:00.006345 18244 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 5 FOLLOWER]: Advancing to term 6
I20250111 08:59:00.010022 18244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250111 08:59:00.613185 18253 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:00.613333 18253 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:00.613914 18253 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:00.614355 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:59:00.614614 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 6.
I20250111 08:59:00.615041 18165 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 
I20250111 08:59:00.615319 18253 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250111 08:59:00.615438 18253 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:59:00.615536 18253 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 6 FOLLOWER]: Advancing to term 7
W20250111 08:59:00.615720 18163 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:00.617511 18253 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:00.618050 18253 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 7 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:00.618356 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e"
I20250111 08:59:00.618566 18047 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 6 FOLLOWER]: Advancing to term 7
W20250111 08:59:00.619184 18163 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:00.620476 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 7.
I20250111 08:59:00.620786 18165 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f
I20250111 08:59:00.621034 18253 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Leader election won for term 7
I20250111 08:59:00.621353 18253 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 LEADER]: Becoming Leader. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Running, Role: LEADER
I20250111 08:59:00.621760 18253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:00.623986 18256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77aa4f8aaf704f1898426e17c6fbe28f. Latest consensus state: current_term: 7 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:00.624204 18256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:00.624776 18258 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:59:00.627085 18258 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:59:00.627511 18258 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:59:00.627857 18258 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:59:00.629597 18258 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:59:00.630450 18258 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:59:00.631275 18258 tablet_loader.cc:96] loaded metadata for tablet c0fa97b400b14819b2a6e69865df8b8c (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:59:00.632097 18258 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:59:00.633235 18258 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:00.633311 18258 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:59:00.634637 18258 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:59:00.635514 18258 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Loaded TSK: 0
I20250111 08:59:00.636300 18258 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:59:00.637208 18258 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:59:00.638109 18258 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:59:00.670058 18177 master_service.cc:946] ReplaceTablet: received request to replace tablet 6634d4c9c6e846ae8c655b59f8d001d8 from {username='slave'} at 127.0.0.1:42572
I20250111 08:59:00.670120 18178 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:42572
I20250111 08:59:00.671176 18176 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.671391 18170 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.671497 18169 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.672487 18178 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.672904 18171 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42572:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:59:00.674168 18177 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.671891 18175 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.674118 18172 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42572:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:59:00.673831 18173 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.673049 18174 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42572:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:00.680171 18232 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:59:00.680588 18232 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:59:00.680662 18232 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
    @     0x7f77d724edcd  google::LogMessage::Fail() at ??:0
    @     0x7f77d7252b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f77d724e7cc  google::LogMessage::Flush() at ??:0
    @     0x7f77d724ea11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f77ed3f16de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
    @     0x7f77ed3f03c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
    @     0x7f77ed3f74dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
    @     0x7f77ed3f7359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
    @     0x7f77d9fac0be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f77dd38f6db  start_thread at ??:0
    @     0x7f77d60ff71f  clone at ??:0
W20250111 08:59:00.869416 18019 connection.cc:537] client connection to 127.16.20.189:35063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 08:59:00.869688 18002 master_runner.cc:186] Error getting master registration for 127.16.20.189:35063: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107), 
I20250111 08:59:00.869846 18002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 08:59:00.870205 17143 connection.cc:537] client connection to 127.16.20.189:35063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:59:00.870703 17399 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.189:35063)
W20250111 08:59:00.871057 17399 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.189:35063)
W20250111 08:59:00.871472 17399 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.189:35063)
I20250111 08:59:01.263350 18273 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:01.263559 18273 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:01.264259 18273 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:59:01.266152 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:01.266333 18018 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:01.266433 18018 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:01.266685 18273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 08:59:02.302803 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--webserver_interface=127.16.20.189
--webserver_port=43097
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 08:59:02.606992 18280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:59:02.607273 18280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:59:02.607398 18280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:59:02.614684 18280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:59:02.614784 18280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:59:02.614854 18280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:59:02.614929 18280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:59:02.623694 18280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.20.189
--webserver_port=43097
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:59:02.625080 18280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:59:02.626608 18280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:59:02.636590 18286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:02.637501 18289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:02.638118 18287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:59:02.638724 18280 server_base.cc:1034] running on GCE node
I20250111 08:59:02.639376 18280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:59:02.640653 18280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:59:02.641852 18280 hybrid_clock.cc:648] HybridClock initialized: now 1736585942641796 us; error 35 us; skew 500 ppm
I20250111 08:59:02.642134 18280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:59:02.644150 18280 webserver.cc:458] Webserver started at http://127.16.20.189:43097/ using document root <none> and password file <none>
I20250111 08:59:02.644845 18280 fs_manager.cc:362] Metadata directory not provided
I20250111 08:59:02.644960 18280 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:59:02.649674 18280 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.003s
I20250111 08:59:02.652184 18295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:59:02.653378 18280 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20250111 08:59:02.653533 18280 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:59:02.654323 18280 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:59:02.677933 18280 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:59:02.678627 18280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:59:02.678843 18280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:59:02.694217 18280 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.189:35063
I20250111 08:59:02.694245 18346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.189:35063 every 8 connection(s)
I20250111 08:59:02.695614 18280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
I20250111 08:59:02.698110 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 18280
I20250111 08:59:02.701817 18347 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:59:02.704845 18347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap starting.
I20250111 08:59:02.715798 18347 log.cc:826] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:59:02.746802 18347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=42 ignored=0} inserts{seen=27 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 08:59:02.747269 18347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap complete.
I20250111 08:59:02.753139 18347 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:02.753990 18347 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Initialized, Role: FOLLOWER
I20250111 08:59:02.754535 18347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 7.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:02.755227 18351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:02.755492 18351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:02.756070 18347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:59:02.766502 18365 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:02.766602 18365 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: loading cluster ID for follower catalog manager: success
I20250111 08:59:02.770841 18365 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: acquiring CA information for follower catalog manager: success
I20250111 08:59:02.773553 18365 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:59:02.791074 18280 master_runner.cc:186] Error getting master registration for 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111), 
I20250111 08:59:02.795810 18280 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:02.879877 17391 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:59:02.881019 17256 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:59:02.882901 18312 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:43623; Asking this server to re-register.
I20250111 08:59:02.882875 18311 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:56087; Asking this server to re-register.
I20250111 08:59:02.883579 17256 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:02.883616 17391 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:02.885274 18312 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:59:02.885353 18311 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:59:02.900807 17118 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:59:02.901844 18312 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:39575; Asking this server to re-register.
I20250111 08:59:02.902411 17118 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:02.903651 18312 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:59:02.946943 18372 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:02.947149 18372 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:02.947810 18372 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:59:02.949333 18018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250111 08:59:02.950757 18018 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:02.955503 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 8 candidate_status { last_received { term: 2 index: 36 } } ignore_live_leader: false dest_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" is_pre_election: true
I20250111 08:59:02.956235 18322 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77238c1f3c9423caf1235b18b67ab2e for term 8 because replica has last-logged OpId of term: 7 index: 43, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 36.
I20250111 08:59:02.956857 18019 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:02.957137 18372 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250111 08:59:03.296314 18376 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:03.296579 18376 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:03.298166 18376 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:03.298689 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:59:03.298906 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 7.
W20250111 08:59:03.298969 18297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:03.299358 18299 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 
I20250111 08:59:03.299719 18376 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250111 08:59:03.299875 18376 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:59:03.299978 18376 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 7 FOLLOWER]: Advancing to term 8
W20250111 08:59:03.300545 18297 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:03.303748 18376 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:03.304293 18376 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 8 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:03.304620 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e"
I20250111 08:59:03.304778 18047 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 7 FOLLOWER]: Advancing to term 8
W20250111 08:59:03.305356 18297 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:03.306697 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 8.
I20250111 08:59:03.307080 18299 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f
I20250111 08:59:03.307380 18376 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 FOLLOWER]: Leader election won for term 8
I20250111 08:59:03.307749 18376 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 LEADER]: Becoming Leader. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Running, Role: LEADER
I20250111 08:59:03.308128 18376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 7.43, Last appended by leader: 43, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:03.310681 18379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77aa4f8aaf704f1898426e17c6fbe28f. Latest consensus state: current_term: 8 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:03.310895 18379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:03.311450 18381 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:59:03.313779 18381 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:59:03.314173 18381 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:59:03.314474 18381 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:59:03.316161 18381 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:59:03.317057 18381 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:59:03.317845 18381 tablet_loader.cc:96] loaded metadata for tablet c0fa97b400b14819b2a6e69865df8b8c (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:59:03.318782 18381 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:59:03.319998 18381 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:03.320173 18381 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:59:03.321458 18381 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:59:03.322289 18381 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Loaded TSK: 0
I20250111 08:59:03.323009 18381 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:59:03.323854 18381 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:59:03.324779 18381 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:59:03.392266 18310 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:44086
I20250111 08:59:03.392385 18312 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:44086
I20250111 08:59:03.393316 18309 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.393592 18303 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.393723 18312 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.394205 18311 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.394330 18308 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.394977 18306 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44086:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:59:03.395175 18310 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.395303 18305 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44086:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:59:03.395679 18307 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.395087 18304 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:03.400686 18364 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:59:03.401072 18364 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:59:03.401160 18364 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
    @     0x7f0f0b27cdcd  google::LogMessage::Fail() at ??:0
    @     0x7f0f0b280b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f0f0b27c7cc  google::LogMessage::Flush() at ??:0
    @     0x7f0f0b27ca11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f0f2141f6de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
    @     0x7f0f2141e3c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
    @     0x7f0f214254dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
    @     0x7f0f21425359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
    @     0x7f0f0dfda0be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f0f113bd6db  start_thread at ??:0
    @     0x7f0f0a12d71f  clone at ??:0
W20250111 08:59:03.573526 17399 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.189:35063)
W20250111 08:59:03.573983 17399 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.189:35063)
W20250111 08:59:03.574448 17399 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.189:35063)
I20250111 08:59:03.889055 17391 heartbeater.cc:502] Master 127.16.20.189:35063 was elected leader, sending a full tablet report...
I20250111 08:59:03.889142 17256 heartbeater.cc:502] Master 127.16.20.189:35063 was elected leader, sending a full tablet report...
I20250111 08:59:05.760293 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--webserver_interface=127.16.20.188
--webserver_port=41249
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 08:59:05.872440 18002 master_runner.cc:186] Error getting master registration for 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111), 
I20250111 08:59:05.872583 18002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 08:59:06.066687 18403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:59:06.066962 18403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:59:06.067085 18403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:59:06.074347 18403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:59:06.074447 18403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:59:06.074518 18403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:59:06.074591 18403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:59:06.084175 18403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.20.188
--webserver_port=41249
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:59:06.085661 18403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:59:06.087267 18403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:59:06.097278 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:06.097842 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:06.098371 18412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:59:06.099426 18403 server_base.cc:1034] running on GCE node
I20250111 08:59:06.100039 18403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:59:06.101246 18403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:59:06.102440 18403 hybrid_clock.cc:648] HybridClock initialized: now 1736585946102389 us; error 26 us; skew 500 ppm
I20250111 08:59:06.102727 18403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:59:06.108839 18403 webserver.cc:458] Webserver started at http://127.16.20.188:41249/ using document root <none> and password file <none>
I20250111 08:59:06.109504 18403 fs_manager.cc:362] Metadata directory not provided
I20250111 08:59:06.109613 18403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:59:06.114369 18403 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.003s
I20250111 08:59:06.117002 18418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:59:06.118216 18403 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20250111 08:59:06.118363 18403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
format_stamp: "Formatted at 2025-01-11 08:58:40 on dist-test-slave-s924"
I20250111 08:59:06.119135 18403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:59:06.159842 18403 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:59:06.160689 18403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:59:06.160921 18403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:59:06.176586 18403 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.188:41785
I20250111 08:59:06.176606 18469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.188:41785 every 8 connection(s)
I20250111 08:59:06.177898 18403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
I20250111 08:59:06.183619 18470 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:59:06.186597 18470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap starting.
I20250111 08:59:06.187690 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 18403
I20250111 08:59:06.201923 18470 log.cc:826] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:59:06.212942 17389 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:59:06.236450 18470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=43 overwritten=0 applied=42 ignored=0} inserts{seen=27 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 08:59:06.236886 18470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap complete.
I20250111 08:59:06.242708 18470 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:06.243551 18470 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Initialized, Role: FOLLOWER
I20250111 08:59:06.244067 18470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 6.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:06.244767 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:06.245005 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:06.245659 18470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:59:06.271721 18489 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:06.271816 18489 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: loading cluster ID for follower catalog manager: success
I20250111 08:59:06.275794 18489 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: acquiring CA information for follower catalog manager: success
I20250111 08:59:06.284332 18489 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:59:06.289376 18403 master_runner.cc:186] Error getting master registration for 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111), 
I20250111 08:59:06.294066 18403 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:06.367225 17254 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:59:06.367417 17254 heartbeater.cc:502] Master 127.16.20.188:41785 was elected leader, sending a full tablet report...
I20250111 08:59:06.369448 18435 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:40605; Asking this server to re-register.
I20250111 08:59:06.369982 17254 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:06.371351 18435 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:59:06.468645 18496 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:06.468815 18496 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:59:06.469607 18496 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 08:59:06.470469 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:59:06.470664 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:59:06.470781 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 08:59:06.471006 18496 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:59:06.569268 17116 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:59:06.570430 18435 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:50647; Asking this server to re-register.
I20250111 08:59:06.570933 17116 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:06.572104 18435 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:59:06.715859 18435 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:44521; Asking this server to re-register.
I20250111 08:59:06.716450 17389 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:06.717542 18435 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:59:06.847568 18497 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:06.847798 18497 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:06.849537 18497 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:59:06.849902 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:59:06.850193 18047 raft_consensus.cc:2363] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f for earlier term 7. Current term is 8.
W20250111 08:59:06.850308 18421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:06.850813 18422 leader_election.cc:400] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 7 pre-election: Vote denied by peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f for earlier term 7. Current term is 8.
I20250111 08:59:06.850890 18422 leader_election.cc:403] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20250111 08:59:06.851200 18497 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 6 FOLLOWER]: Advancing to term 8
W20250111 08:59:06.851982 18421 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:06.854854 18497 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f for earlier term 7. Current term is 8.
I20250111 08:59:07.480562 18506 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:07.480720 18506 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:07.481385 18506 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:59:07.481895 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 9 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:59:07.482133 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 8.
I20250111 08:59:07.482544 18422 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 
I20250111 08:59:07.483000 18506 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250111 08:59:07.483266 18506 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:59:07.483430 18506 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 8 FOLLOWER]: Advancing to term 9
W20250111 08:59:07.483948 18421 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:07.485603 18506 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:07.486132 18506 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 9 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:59:07.486564 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 9 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e"
I20250111 08:59:07.486778 18047 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 8 FOLLOWER]: Advancing to term 9
W20250111 08:59:07.487423 18421 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:07.488826 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in term 9.
I20250111 08:59:07.489171 18422 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:07.489454 18506 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 FOLLOWER]: Leader election won for term 9
I20250111 08:59:07.489799 18506 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 LEADER]: Becoming Leader. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Running, Role: LEADER
I20250111 08:59:07.490226 18506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 6.43, Last appended by leader: 43, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:07.492776 18509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb4ec5faa3472f9cecb23f9b5d2e0f. Latest consensus state: current_term: 9 leader_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:07.493046 18509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:07.493597 18511 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:59:07.496080 18511 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:59:07.496526 18511 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:59:07.496832 18511 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:59:07.498481 18511 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:59:07.499259 18511 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:59:07.500005 18511 tablet_loader.cc:96] loaded metadata for tablet c0fa97b400b14819b2a6e69865df8b8c (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:59:07.500849 18511 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:59:07.501721 18511 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:07.501785 18511 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:59:07.502929 18511 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:59:07.503752 18511 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Loaded TSK: 0
I20250111 08:59:07.504562 18511 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:59:07.505407 18511 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:59:07.506310 18511 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:59:07.528982 18435 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:35038
I20250111 08:59:07.529012 18434 master_service.cc:946] ReplaceTablet: received request to replace tablet 6634d4c9c6e846ae8c655b59f8d001d8 from {username='slave'} at 127.0.0.1:35038
I20250111 08:59:07.529987 18429 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35038:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:59:07.529755 18433 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.530649 18434 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.530933 18435 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.531044 18426 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.531780 18427 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.531936 18431 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.531811 18428 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35038:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:59:07.532315 18430 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.532470 18432 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35038:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:07.537082 18488 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:59:07.537431 18488 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:59:07.537497 18488 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
    @     0x7fc8f5b73dcd  google::LogMessage::Fail() at ??:0
    @     0x7fc8f5b77b93  google::LogMessage::SendToLog() at ??:0
    @     0x7fc8f5b737cc  google::LogMessage::Flush() at ??:0
    @     0x7fc8f5b73a11  google::LogMessage::~LogMessage() at ??:0
    @     0x7fc90bd166de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
    @     0x7fc90bd153c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
    @     0x7fc90bd1c4dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
    @     0x7fc90bd1c359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
    @     0x7fc8f88d10be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fc8fbcb46db  start_thread at ??:0
    @     0x7fc8f4a2471f  clone at ??:0
W20250111 08:59:07.711730 17004 connection.cc:537] client connection to 127.16.20.188:41785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:59:07.712553 17398 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.188:41785)
W20250111 08:59:07.712930 17398 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.188:41785)
W20250111 08:59:07.713445 17398 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.188:41785)
I20250111 08:59:08.901407 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--webserver_interface=127.16.20.189
--webserver_port=43097
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250111 08:59:09.206465 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:59:09.206732 18529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:59:09.206852 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:59:09.214687 18529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:59:09.214802 18529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:59:09.214867 18529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:59:09.214924 18529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:59:09.225428 18529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.189:35063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.20.189
--webserver_port=43097
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:59:09.227305 18529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:59:09.229235 18529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:59:09.239457 18536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:09.240319 18535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:09.240991 18538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:59:09.241598 18529 server_base.cc:1034] running on GCE node
I20250111 08:59:09.242285 18529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:59:09.243510 18529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:59:09.244711 18529 hybrid_clock.cc:648] HybridClock initialized: now 1736585949244650 us; error 33 us; skew 500 ppm
I20250111 08:59:09.245011 18529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:59:09.247094 18529 webserver.cc:458] Webserver started at http://127.16.20.189:43097/ using document root <none> and password file <none>
I20250111 08:59:09.247749 18529 fs_manager.cc:362] Metadata directory not provided
I20250111 08:59:09.247859 18529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:59:09.252640 18529 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20250111 08:59:09.255174 18544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:59:09.256327 18529 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20250111 08:59:09.256518 18529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
uuid: "77aa4f8aaf704f1898426e17c6fbe28f"
format_stamp: "Formatted at 2025-01-11 08:58:39 on dist-test-slave-s924"
I20250111 08:59:09.257277 18529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:59:09.281525 18529 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:59:09.282267 18529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:59:09.282518 18529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:59:09.297408 18529 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.189:35063
I20250111 08:59:09.297436 18595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.189:35063 every 8 connection(s)
I20250111 08:59:09.298692 18529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-1/data/info.pb
I20250111 08:59:09.304242 18596 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:59:09.306792 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 18529
I20250111 08:59:09.307138 18596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap starting.
I20250111 08:59:09.318269 18596 log.cc:826] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:59:09.350044 18596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=42 ignored=0} inserts{seen=27 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250111 08:59:09.350521 18596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Bootstrap complete.
I20250111 08:59:09.356462 18596 raft_consensus.cc:357] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:09.357324 18596 raft_consensus.cc:738] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Initialized, Role: FOLLOWER
I20250111 08:59:09.357873 18596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 8.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:09.358552 18600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:09.358808 18600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:09.359431 18596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:59:09.379655 18614 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:09.380190 18614 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: loading cluster ID for follower catalog manager: success
I20250111 08:59:09.385094 18614 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: acquiring CA information for follower catalog manager: success
I20250111 08:59:09.388737 18614 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:59:09.399724 18529 master_runner.cc:186] Error getting master registration for 127.16.20.188:41785: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111), 
I20250111 08:59:09.406606 18529 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:09.408736 17256 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:59:09.408908 17256 heartbeater.cc:502] Master 127.16.20.189:35063 was elected leader, sending a full tablet report...
I20250111 08:59:09.411787 18561 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:44295; Asking this server to re-register.
I20250111 08:59:09.413872 17256 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:09.415437 18561 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:59:09.416414 17391 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:59:09.416616 17391 heartbeater.cc:502] Master 127.16.20.189:35063 was elected leader, sending a full tablet report...
I20250111 08:59:09.417941 18561 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:58425; Asking this server to re-register.
I20250111 08:59:09.418473 17391 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:09.419562 18561 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:59:09.589920 17118 heartbeater.cc:346] Connected to a master server at 127.16.20.189:35063
I20250111 08:59:09.591238 18561 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:42035; Asking this server to re-register.
I20250111 08:59:09.591779 17118 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:09.593050 18561 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:59:09.734961 18600 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:09.735210 18600 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:09.737051 18600 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:09.737476 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
W20250111 08:59:09.737646 18546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:09.737743 18047 raft_consensus.cc:2388] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77aa4f8aaf704f1898426e17c6fbe28f in current term 9: Already voted for candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in this term.
W20250111 08:59:09.739059 18546 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:09.739216 18546 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:09.739547 18600 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 8 FOLLOWER]: Advancing to term 9
I20250111 08:59:09.744048 18600 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250111 08:59:10.315559 18625 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:10.315716 18625 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:10.316334 18625 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:10.316710 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 10 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:59:10.316906 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 9.
I20250111 08:59:10.317284 18548 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 
I20250111 08:59:10.317584 18625 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250111 08:59:10.317723 18625 raft_consensus.cc:491] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 08:59:10.317823 18625 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 9 FOLLOWER]: Advancing to term 10
W20250111 08:59:10.318222 18546 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:10.319783 18625 raft_consensus.cc:513] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:10.320286 18625 leader_election.cc:290] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 10 election: Requested vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:10.320662 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" candidate_term: 10 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e"
I20250111 08:59:10.320860 18047 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 9 FOLLOWER]: Advancing to term 10
W20250111 08:59:10.321400 18546 leader_election.cc:336] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:10.322629 18047 raft_consensus.cc:2463] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77aa4f8aaf704f1898426e17c6fbe28f in term 10.
I20250111 08:59:10.322947 18548 leader_election.cc:304] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f
I20250111 08:59:10.323200 18625 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 10 FOLLOWER]: Leader election won for term 10
I20250111 08:59:10.323525 18625 raft_consensus.cc:695] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [term 10 LEADER]: Becoming Leader. State: Replica: 77aa4f8aaf704f1898426e17c6fbe28f, State: Running, Role: LEADER
I20250111 08:59:10.323968 18625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 8.44, Last appended by leader: 44, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:10.326491 18628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77aa4f8aaf704f1898426e17c6fbe28f. Latest consensus state: current_term: 10 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:10.326722 18628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:10.327255 18630 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:59:10.329533 18630 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:59:10.329945 18630 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:59:10.330287 18630 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:59:10.331954 18630 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:59:10.332840 18630 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:59:10.333674 18630 tablet_loader.cc:96] loaded metadata for tablet c0fa97b400b14819b2a6e69865df8b8c (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:59:10.334479 18630 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:59:10.335410 18630 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:10.335475 18630 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:59:10.336694 18630 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:59:10.337560 18630 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f: Loaded TSK: 0
I20250111 08:59:10.338331 18630 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:59:10.339185 18630 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:59:10.340106 18630 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:59:10.369788 18613 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:59:10.370196 18613 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:59:10.370282 18613 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
    @     0x7f87b770edcd  google::LogMessage::Fail() at ??:0
    @     0x7f87b7712b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f87b770e7cc  google::LogMessage::Flush() at ??:0
    @     0x7f87b770ea11  google::LogMessage::~LogMessage() at ??:0
I20250111 08:59:10.378115 18561 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:45480
I20250111 08:59:10.378407 18560 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:45480
I20250111 08:59:10.379168 18557 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:10.381986 18561 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:10.381254 18556 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:10.383116 18553 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
    @     0x7f87cd8b16de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250111 08:59:10.384224 18554 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45480:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:59:10.382846 18555 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45480:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:59:10.382959 18558 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:10.385434 18552 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:10.385632 18559 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:10.386270 18560 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45480:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
    @     0x7f87cd8b03c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
    @     0x7f87cd8b74dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
    @     0x7f87cd8b7359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
I20250111 08:59:10.399564 18047 raft_consensus.cc:1270] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Refusing update from remote peer 77aa4f8aaf704f1898426e17c6fbe28f: Log matching property violated. Preceding OpId in replica: term: 2 index: 36. Preceding OpId from leader: term: 10 index: 46. (index mismatch)
W20250111 08:59:10.400071 18546 consensus_peers.cc:487] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f -> Peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Couldn't send request to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f. Status: Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 08:59:10.400283 18628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77aa4f8aaf704f1898426e17c6fbe28f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 36, Time since last communication: 0.000s
    @     0x7f87ba46c0be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f87bd84f6db  start_thread at ??:0
    @     0x7f87b65bf71f  clone at ??:0
I20250111 08:59:10.414495 18648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:10.414847 18648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:10.415118 18642 mvcc.cc:204] Tried to move back new op lower bound from 7113055979246825472 to 7113055949065019392. Current Snapshot: MvccSnapshot[applied={T|T < 7113055979246825472 or (T in {7113055979246825472})}]
I20250111 08:59:10.415200 18645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:10.415249 18641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77aa4f8aaf704f1898426e17c6fbe28f. Latest consensus state: current_term: 10 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:10.415524 18645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:10.415580 18641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:10.415668 18650 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:10.415849 18650 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: FOLLOWER
W20250111 08:59:10.558179 17143 connection.cc:537] client connection to 127.16.20.189:35063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:59:10.558230 17279 connection.cc:537] client connection to 127.16.20.189:35063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:59:10.558260 18018 connection.cc:537] server connection from 127.0.0.1:41156 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:59:10.559278 17399 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.189:35063)
W20250111 08:59:10.559644 17399 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.189:35063)
W20250111 08:59:10.560112 17399 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.189:35063)
I20250111 08:59:10.874905 18002 master_runner.cc:186] Error getting master registration for 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111), 
I20250111 08:59:10.875072 18002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:10.904119 18650 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:59:10.904281 18650 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:10.904978 18650 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:59:10.905345 18019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:10.906729 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:10.906883 18018 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:10.907016 18018 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:10.907263 18650 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250111 08:59:11.507517 18660 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:59:11.507683 18660 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:11.508276 18660 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:59:11.510000 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:11.510134 18018 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:11.510273 18018 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:11.510509 18660 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250111 08:59:12.483448 16466 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2kHLUS/build/asan/bin/kudu
/tmp/dist-test-task2kHLUS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--webserver_interface=127.16.20.188
--webserver_port=41249
--builtin_ntp_servers=127.16.20.148:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250111 08:59:12.499212 18667 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:59:12.499356 18667 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:12.499969 18667 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:59:12.501785 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:12.502068 18018 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785): Network error: Client connection negotiation failed: client connection to 127.16.20.188:41785: connect: Connection refused (error 111)
I20250111 08:59:12.502197 18018 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77238c1f3c9423caf1235b18b67ab2e; no voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:12.502431 18667 raft_consensus.cc:2743] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250111 08:59:12.789690 18666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 08:59:12.789945 18666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 08:59:12.790052 18666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 08:59:12.797453 18666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 08:59:12.797541 18666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 08:59:12.797590 18666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 08:59:12.797644 18666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 08:59:12.806926 18666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.20.148:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45979
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.20.188:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.20.188
--webserver_port=41249
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 08:43:21 UTC on 24a791456cd2
build id 4123
ASAN enabled
I20250111 08:59:12.808344 18666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 08:59:12.809921 18666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 08:59:12.820032 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:12.820612 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 08:59:12.821514 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 08:59:12.822283 18666 server_base.cc:1034] running on GCE node
I20250111 08:59:12.822919 18666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 08:59:12.824134 18666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 08:59:12.825333 18666 hybrid_clock.cc:648] HybridClock initialized: now 1736585952825252 us; error 55 us; skew 500 ppm
I20250111 08:59:12.825632 18666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 08:59:12.832444 18666 webserver.cc:458] Webserver started at http://127.16.20.188:41249/ using document root <none> and password file <none>
I20250111 08:59:12.833087 18666 fs_manager.cc:362] Metadata directory not provided
I20250111 08:59:12.833201 18666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 08:59:12.838084 18666 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.003s
I20250111 08:59:12.840584 18684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20250111 08:59:12.841699 18666 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20250111 08:59:12.841835 18666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data,/tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
format_stamp: "Formatted at 2025-01-11 08:58:40 on dist-test-slave-s924"
I20250111 08:59:12.842594 18666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 08:59:12.868721 18666 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20250111 08:59:12.869637 18666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 08:59:12.870594 18666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 08:59:12.886554 18666 rpc_server.cc:307] RPC server started. Bound to: 127.16.20.188:41785
I20250111 08:59:12.886994 18736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.20.188:41785 every 8 connection(s)
I20250111 08:59:12.888396 18666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2kHLUS/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1736585905530747-16466-0/minicluster-data/master-2/data/info.pb
I20250111 08:59:12.889231 16466 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2kHLUS/build/asan/bin/kudu as pid 18666
I20250111 08:59:12.895387 18738 sys_catalog.cc:263] Verifying existing consensus state
I20250111 08:59:12.898674 18738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap starting.
I20250111 08:59:12.907341 17254 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:59:12.911792 18738 log.cc:826] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Log is configured to *not* fsync() on all Append() calls
I20250111 08:59:12.944073 18738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=42 ignored=0} inserts{seen=27 ignored=0} mutations{seen=28 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250111 08:59:12.944761 18738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Bootstrap complete.
I20250111 08:59:12.950938 18738 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:12.951857 18738 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acb4ec5faa3472f9cecb23f9b5d2e0f, State: Initialized, Role: FOLLOWER
I20250111 08:59:12.952458 18738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 9.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:12.953156 18742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:12.953393 18742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:12.954097 18738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: configured and running, proceeding with master startup.
I20250111 08:59:12.977032 18756 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:12.977124 18756 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: loading cluster ID for follower catalog manager: success
I20250111 08:59:12.980710 18756 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: acquiring CA information for follower catalog manager: success
I20250111 08:59:12.983047 18756 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 08:59:12.986117 18666 master_runner.cc:186] Error getting master registration for 127.16.20.189:35063: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111), 
I20250111 08:59:12.990821 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:13.229473 17116 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:59:13.231930 18702 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" instance_seqno: 1736585921254744) as {username='slave'} at 127.16.20.129:45025; Asking this server to re-register.
I20250111 08:59:13.232590 17116 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:13.234414 18702 ts_manager.cc:194] Registered new tserver with Master: 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699)
I20250111 08:59:13.239451 17389 heartbeater.cc:346] Connected to a master server at 127.16.20.188:41785
I20250111 08:59:13.240537 18702 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "25e23fc83774472986af5c3f81f591fe" instance_seqno: 1736585922172878) as {username='slave'} at 127.16.20.131:51805; Asking this server to re-register.
I20250111 08:59:13.240981 17389 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:13.242089 18702 ts_manager.cc:194] Registered new tserver with Master: 25e23fc83774472986af5c3f81f591fe (127.16.20.131:33627)
I20250111 08:59:13.410414 18702 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" instance_seqno: 1736585921716710) as {username='slave'} at 127.16.20.130:33127; Asking this server to re-register.
I20250111 08:59:13.410995 17254 heartbeater.cc:463] Registering TS with master...
I20250111 08:59:13.412082 18702 ts_manager.cc:194] Registered new tserver with Master: d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
I20250111 08:59:13.509197 18762 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:13.509446 18762 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:13.511183 18762 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
I20250111 08:59:13.511693 18047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "c77238c1f3c9423caf1235b18b67ab2e" is_pre_election: true
I20250111 08:59:13.511941 18047 raft_consensus.cc:2388] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4acb4ec5faa3472f9cecb23f9b5d2e0f in current term 10: Already voted for candidate 77aa4f8aaf704f1898426e17c6fbe28f in this term.
W20250111 08:59:13.512012 18687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:13.513460 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:13.513612 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:13.513917 18762 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 9 FOLLOWER]: Advancing to term 10
I20250111 08:59:13.517614 18762 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250111 08:59:14.140609 18770 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:59:14.140786 18770 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:14.141402 18770 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
W20250111 08:59:14.143167 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:14.147410 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 46 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" is_pre_election: true
I20250111 08:59:14.147809 18712 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77238c1f3c9423caf1235b18b67ab2e in term 10.
I20250111 08:59:14.148192 18018 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:14.148465 18770 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250111 08:59:14.148571 18770 raft_consensus.cc:491] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 77aa4f8aaf704f1898426e17c6fbe28f)
I20250111 08:59:14.148659 18770 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 10 FOLLOWER]: Advancing to term 11
I20250111 08:59:14.150543 18770 raft_consensus.cc:513] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:14.151036 18770 leader_election.cc:290] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 election: Requested vote from peers 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063), 4acb4ec5faa3472f9cecb23f9b5d2e0f (127.16.20.188:41785)
I20250111 08:59:14.151504 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77238c1f3c9423caf1235b18b67ab2e" candidate_term: 11 candidate_status { last_received { term: 10 index: 46 } } ignore_live_leader: false dest_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f"
I20250111 08:59:14.151701 18712 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 10 FOLLOWER]: Advancing to term 11
W20250111 08:59:14.152086 18019 leader_election.cc:336] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:14.153757 18712 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77238c1f3c9423caf1235b18b67ab2e in term 11.
I20250111 08:59:14.154078 18018 leader_election.cc:304] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f, c77238c1f3c9423caf1235b18b67ab2e; no voters: 77aa4f8aaf704f1898426e17c6fbe28f
I20250111 08:59:14.154314 18770 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 11 FOLLOWER]: Leader election won for term 11
I20250111 08:59:14.154704 18770 raft_consensus.cc:695] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [term 11 LEADER]: Becoming Leader. State: Replica: c77238c1f3c9423caf1235b18b67ab2e, State: Running, Role: LEADER
I20250111 08:59:14.155169 18770 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 10.46, Last appended by leader: 46, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:14.156689 18775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77238c1f3c9423caf1235b18b67ab2e. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.156997 18775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:14.598340 18712 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Refusing update from remote peer c77238c1f3c9423caf1235b18b67ab2e: Log matching property violated. Preceding OpId in replica: term: 9 index: 44. Preceding OpId from leader: term: 11 index: 47. (index mismatch)
I20250111 08:59:14.599014 18775 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 42, Time since last communication: 0.000s
I20250111 08:59:14.599246 18775 consensus_queue.cc:1225] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [LEADER]: Peer 4acb4ec5faa3472f9cecb23f9b5d2e0f log is divergent from this leader: its last log entry 9.44 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20250111 08:59:14.600509 18712 pending_rounds.cc:77] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Aborting all ops after (but not including) 42
I20250111 08:59:14.600625 18712 pending_rounds.cc:99] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Aborting uncommitted NO_OP operation due to leader change: 6.43
I20250111 08:59:14.600725 18712 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 08:59:14.600889 18712 pending_rounds.cc:99] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Aborting uncommitted NO_OP operation due to leader change: 9.44
I20250111 08:59:14.600946 18712 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 08:59:14.605369 18781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77238c1f3c9423caf1235b18b67ab2e. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.605660 18781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:14.612756 18786 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.613144 18786 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:14.613337 18775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.613153 18787 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.613687 18787 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:14.613598 18775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: LEADER
I20250111 08:59:14.614295 18775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.614523 18775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e [sys.catalog]: This master's current role is: LEADER
W20250111 08:59:14.614622 18019 consensus_peers.cc:487] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e -> Peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Couldn't send request to peer 77aa4f8aaf704f1898426e17c6fbe28f. Status: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 08:59:14.615330 18781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.615564 18781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:14.615496 18791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.615787 18791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:14.615916 18788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.615938 18793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "c77238c1f3c9423caf1235b18b67ab2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } } }
I20250111 08:59:14.616243 18788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:14.616266 18793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [sys.catalog]: This master's current role is: FOLLOWER
I20250111 08:59:14.672708 18777 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 08:59:14.675318 18777 catalog_manager.cc:670] Loaded metadata for table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6]
I20250111 08:59:14.675794 18777 catalog_manager.cc:670] Loaded metadata for table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a]
I20250111 08:59:14.676157 18777 catalog_manager.cc:670] Loaded metadata for table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb]
I20250111 08:59:14.677981 18777 tablet_loader.cc:96] loaded metadata for tablet 30dabd99c09f4a0099a111c8e8512f52 (table default.table_971c3c3dd7714c618c4d251288537810 [id=d969916042a845fcb0281c184eecab5a])
I20250111 08:59:14.678901 18777 tablet_loader.cc:96] loaded metadata for tablet 9c0e4e00771d432ea7467fc2082c4518 (table default.table_4b95455ed62a4f51a8aa85b327bc4699 [id=f0ca1fa5395c49b7b84a8837644895fb])
I20250111 08:59:14.679823 18777 tablet_loader.cc:96] loaded metadata for tablet c0fa97b400b14819b2a6e69865df8b8c (table default.table_4838c339e9c747e1b906df07efeba1bc [id=5965ede942dd4486a987eee2c8fc4ad6])
I20250111 08:59:14.680727 18777 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 08:59:14.681645 18777 catalog_manager.cc:1260] Loaded cluster ID: f3a74dcd1f1a498a837ed85aa2e826e4
I20250111 08:59:14.681715 18777 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 08:59:14.682911 18777 catalog_manager.cc:1505] Loading token signing keys...
I20250111 08:59:14.683770 18777 catalog_manager.cc:5910] T 00000000000000000000000000000000 P c77238c1f3c9423caf1235b18b67ab2e: Loaded TSK: 0
I20250111 08:59:14.684620 18777 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 08:59:14.685503 18777 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250111 08:59:14.686394 18777 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 16
I20250111 08:59:14.727129 18036 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:52620
I20250111 08:59:14.728459 18036 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_518dba2592a54eddbd2b2caec0bb4cfa"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.728817 18035 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_a0ad18d08e114a01ac92b944b0d9efec"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.729564 18034 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_413c709bdd944e2d98cb0e89f0a029ea"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.730172 18033 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_da4abbdb96ec46e1a5a37a5845eeb580"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.730754 18032 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_047f7156c794468f8b656960fc0edb9b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.731491 18031 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52620:
table { table_name: "default.Table_bd323ab641234f2ebfe5a90526f93cdd" } new_table_name: "default.Table_4838c339e9c747e1b906df07efeba1bc" modify_external_catalogs: true
I20250111 08:59:14.731981 18030 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52620:
table { table_name: "default.Table_971c3c3dd7714c618c4d251288537810" } new_table_name: "default.Table_de2fefa7d9d048e6bd99e3d3d7813d9e" modify_external_catalogs: true
I20250111 08:59:14.732488 18029 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_137997c870c04b4e95e9ceb8106ce570"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.733101 18028 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.733954 18037 master_service.cc:946] ReplaceTablet: received request to replace tablet b43238f36aa44f87b625287daef14f76 from {username='slave'} at 127.0.0.1:52620
I20250111 08:59:14.734505 18037 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52620:
name: "default.Table_5f3d49b148aa4b76b52aff24d64f1ad6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20250111 08:59:14.739957 18090 catalog_manager.cc:2710] Deleting table default.table_1c85c3b17bfe4badafe08b8a5b0ce91b [id=f00998fbd79b4a5baec97891252d6a98] in response to Hive Metastore notification log event 17
W20250111 08:59:14.740412 18090 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 17 DROP_TABLE default.table_1c85c3b17bfe4badafe08b8a5b0ce91b: Not found: the table does not exist: table_id: "f00998fbd79b4a5baec97891252d6a98" table_name: "default.table_1c85c3b17bfe4badafe08b8a5b0ce91b"
F20250111 08:59:14.740500 18090 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 17): 17 ALTER_TABLE default.table_de2fefa7d9d048e6bd99e3d3d7813d9e
*** Check failure stack trace: ***
    @     0x7f36db290dcd  google::LogMessage::Fail() at ??:0
    @     0x7f36db294b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f36db2907cc  google::LogMessage::Flush() at ??:0
    @     0x7f36db290a11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f36f14336de  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
    @     0x7f36f14323c1  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
    @     0x7f36f14394dc  kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
    @     0x7f36f1439359  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa71ecd  std::function<>::operator()() at ??:?
    @     0x7f36ddfee0be  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f36e13d16db  start_thread at ??:0
    @     0x7f36da14171f  clone at ??:0
W20250111 08:59:14.924156 17144 connection.cc:537] client connection to 127.16.20.190:46487 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 08:59:14.925462 17400 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.190:46487)
W20250111 08:59:14.925863 17400 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.20.190:46487)
W20250111 08:59:14.926239 17400 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.16.20.190:46487)
I20250111 08:59:15.066643 16466 master-stress-test.cc:577] Killing test thread 1/25
I20250111 08:59:15.107136 18793 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:15.107290 18793 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:15.107864 18793 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:15.109622 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:15.109757 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:15.109889 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:15.110177 18793 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:15.223138 17390 heartbeater.cc:502] Master 127.16.20.190:46487 was elected leader, sending a full tablet report...
I20250111 08:59:15.713994 18813 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:15.714201 18813 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:15.714833 18813 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:15.716439 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:15.716590 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:15.716709 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:15.716910 18813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:16.340556 18818 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:16.340760 18818 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:16.341360 18818 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:16.343004 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:16.343135 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:16.343282 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:16.343497 18818 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:17.849638 18826 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:17.849830 18826 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:17.850432 18826 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:17.852198 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:17.852430 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:59:17.852560 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:17.852818 18826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:17.991981 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:17.992167 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:20.372642 18835 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:20.372854 18835 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:20.373489 18835 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:20.376981 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250111 08:59:20.378008 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:20.378163 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:59:20.378283 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:20.378522 18835 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:21.476095 18840 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:21.476269 18840 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:59:21.477165 18840 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 08:59:21.477895 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:59:21.478044 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:59:21.478150 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 08:59:21.478350 18840 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:59:22.390450 18842 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:22.390633 18842 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:22.391237 18842 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:22.393054 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:22.393213 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:22.393323 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:22.393535 18842 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:22.993284 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:22.993441 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:25.316222 18853 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:25.316411 18853 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:59:25.317021 18853 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 08:59:25.317757 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:59:25.317914 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:59:25.318043 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 08:59:25.318260 18853 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:59:27.994776 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:27.994947 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:28.088550 18861 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:28.088758 18861 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:28.089421 18861 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:28.089982 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 08:59:28.091184 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:28.091387 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:59:28.091492 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:28.091713 18861 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:29.285880 18866 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:29.286048 18866 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:29.286613 18866 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:29.288427 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:29.288745 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:29.288849 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:29.289093 18866 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:32.996210 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:32.996425 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:37.997622 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:37.997783 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:38.284705 18887 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:38.284860 18887 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:59:38.285581 18887 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 08:59:38.286360 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:59:38.286520 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:59:38.286636 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 08:59:38.286854 18887 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:59:40.035243 18892 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:40.035452 18892 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:40.036173 18892 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:40.037055 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 08:59:40.038393 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:40.038563 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:40.038681 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:40.038928 18892 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250111 08:59:42.781888 17390 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (55 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:42.981053 17255 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (56 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:42.981483 17117 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (56 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:59:42.999037 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:42.999191 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:48.000288 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:48.000476 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:48.054672 18915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:48.054883 18915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:48.055588 18915 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:48.056102 18687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 08:59:48.057381 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:48.057514 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:59:48.057662 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:48.057907 18915 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:49.252553 18920 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:49.252739 18920 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:49.253320 18920 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:49.254962 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 08:59:49.255162 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 08:59:49.255278 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:49.255518 18920 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:53.001888 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:53.002051 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 08:59:55.424954 18939 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 08:59:55.425127 18939 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 08:59:55.425755 18939 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 08:59:55.428951 18687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 08:59:55.430033 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 08:59:55.430258 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 08:59:55.430356 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 08:59:55.430557 18939 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 08:59:56.037796 18937 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 08:59:56.037981 18937 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 08:59:56.038901 18937 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 08:59:56.039832 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 08:59:56.039991 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 08:59:56.040131 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 08:59:56.040390 18937 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 08:59:58.003407 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 08:59:58.003585 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:02.746825 18958 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:00:02.746980 18958 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:00:02.747649 18958 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:00:02.748320 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:00:02.748636 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:00:02.748788 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:00:02.749022 18958 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:00:03.004891 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:03.005071 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:08.006469 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:08.006657 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:12.526800 18978 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:00:12.526959 18978 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:00:12.527735 18978 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:00:12.528509 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:00:12.528826 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:00:12.528939 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:00:12.529142 18978 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:00:13.008132 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:13.008316 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:15.075482 18985 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:00:15.075681 18985 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:00:15.076557 18985 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:00:15.077502 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:00:15.078590 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:00:15.078755 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:00:15.078881 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:00:15.079125 18985 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:00:17.667259 18995 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:00:17.667446 18995 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:00:17.668062 18995 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:00:17.669914 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:00:17.670089 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:00:17.670215 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:00:17.670444 18995 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:00:17.933949 18999 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:00:17.934170 18999 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:00:17.934886 18999 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:00:17.935820 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:00:17.935974 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:00:17.936110 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:00:17.936391 18999 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:00:18.009383 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:18.009554 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:20.562258 19005 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:00:20.562502 19005 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:00:20.563123 19005 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:00:20.563908 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 09:00:20.564999 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:00:20.565152 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:00:20.565274 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:00:20.565528 19005 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:00:23.010763 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:23.010934 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:28.012149 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:28.012318 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:33.013635 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:33.013815 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:34.877573 19043 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:00:34.877736 19043 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:00:34.878558 19043 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:00:34.879547 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:00:34.879705 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:00:34.879815 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:00:34.880049 19043 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:00:38.015098 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:38.015273 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:39.084839 19052 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:00:39.085036 19052 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:00:39.085927 19052 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:00:39.086848 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:00:39.087862 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:00:39.088018 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:00:39.088126 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:00:39.088392 19052 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:00:40.112542 19056 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:00:40.112715 19056 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:00:40.113394 19056 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:00:40.114215 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:00:40.114377 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:00:40.114521 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:00:40.114754 19056 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250111 09:00:42.900600 17390 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (175 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:00:43.016575 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:43.016754 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 09:00:43.109441 17255 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (176 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:00:43.109443 17117 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (176 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:00:47.391824 19071 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:00:47.391999 19071 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:00:47.392694 19071 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:00:47.393452 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:00:47.393610 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:00:47.393741 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:00:47.393959 19071 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:00:47.437978 19073 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:00:47.438155 19073 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:00:47.438820 19073 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:00:47.439648 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:00:47.440788 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:00:47.440958 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:00:47.441063 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:00:47.441293 19073 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:00:48.017930 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:48.018092 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:53.021016 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:53.021175 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:00:55.248912 19094 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:00:55.249102 19094 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:00:55.249738 19094 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:00:55.250696 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:00:55.251835 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:00:55.252002 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:00:55.252138 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:00:55.252394 19094 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:00:58.022496 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:00:58.022670 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:03.023937 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:03.024116 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:06.639791 19120 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:01:06.639989 19120 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:01:06.640901 19120 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:01:06.641793 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:01:06.641952 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:01:06.642091 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:01:06.642335 19120 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:01:08.025348 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:08.025499 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:11.915006 19133 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:01:11.915201 19133 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:01:11.916087 19133 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:01:11.917048 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:01:11.918102 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:01:11.918291 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:01:11.918409 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:01:11.918627 19133 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:01:12.175439 19137 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:01:12.175603 19137 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:01:12.176203 19137 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:01:12.176939 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:01:12.177083 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:01:12.177207 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:01:12.177426 19137 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:01:13.012430 18691 maintenance_manager.cc:418] P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033350
I20250111 09:01:13.049484 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:13.049707 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:13.065917 18690 maintenance_manager.cc:642] P 4acb4ec5faa3472f9cecb23f9b5d2e0f: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.053s	user 0.015s	sys 0.009s Metrics: {"bytes_written":12642,"cfile_init":1,"dirs.queue_time_us":325,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1112,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":23,"thread_start_us":388,"threads_started":2,"wal-append.queue_time_us":269}
I20250111 09:01:13.067016 18691 maintenance_manager.cc:418] P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1406 bytes on disk
I20250111 09:01:13.067643 18690 maintenance_manager.cc:642] P 4acb4ec5faa3472f9cecb23f9b5d2e0f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20250111 09:01:18.051079 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:18.051249 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:18.568958 19152 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:01:18.569145 19152 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:01:18.569758 19152 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:01:18.570616 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:01:18.570796 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:01:18.570894 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:01:18.571094 19152 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:01:19.752516 19156 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:01:19.752709 19156 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:01:19.753430 19156 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:01:19.754357 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:01:19.756023 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:01:19.756196 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:01:19.756294 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:01:19.756533 19156 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:01:23.052814 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:23.052983 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:28.054394 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:28.054577 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:28.722846 19212 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:01:28.723065 19212 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:01:28.724138 19212 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:01:28.724892 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:01:28.725135 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:01:28.725242 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:01:28.725492 19212 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:01:29.212002 19215 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:01:29.212188 19215 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:01:29.212843 19215 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:01:29.213697 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:01:29.215031 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:01:29.215169 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:01:29.215296 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:01:29.215524 19215 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:01:33.055868 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:33.056041 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:34.407627 19247 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:01:34.407796 19247 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:01:34.408475 19247 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:01:34.409175 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:01:34.409359 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:01:34.409482 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:01:34.409705 19247 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:01:37.198279 19267 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:01:37.198457 19267 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:01:37.199055 19267 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:01:37.199844 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:01:37.201139 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:01:37.201700 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:01:37.201836 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:01:37.202085 19267 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:01:38.057463 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:38.057655 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:38.474491 19279 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:01:38.474726 19279 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:01:38.475513 19279 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:01:38.477314 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:01:38.477474 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:01:38.477624 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:01:38.477864 19279 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20250111 09:01:43.020229 17390 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (295 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:01:43.060782 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:43.060966 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 09:01:43.230865 17255 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (296 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:01:43.230850 17117 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (296 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:01:48.062284 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:48.062451 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:50.945683 19353 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:01:50.945860 19353 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:01:50.946611 19353 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:01:50.947417 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:01:50.947583 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:01:50.947729 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:01:50.947970 19353 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:01:53.063742 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:53.063912 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:53.810612 19370 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:01:53.810837 19370 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:01:53.811693 19370 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:01:53.812541 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 09:01:53.813546 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:01:53.813709 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:01:53.813831 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:01:53.814054 19370 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:01:58.065199 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:01:58.065368 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:01:58.084831 19399 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:01:58.085014 19399 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:01:58.085669 19399 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:01:58.087186 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:01:58.087375 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:01:58.087493 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:01:58.087730 19399 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:02:03.068519 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:03.068696 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:05.891283 19448 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:02:05.891477 19448 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:02:05.892165 19448 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:02:05.893019 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 09:02:05.894062 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:02:05.894220 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:02:05.894340 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:02:05.894587 19448 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:02:06.262710 19444 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:02:06.262894 19444 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:02:06.263797 19444 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:02:06.264693 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:02:06.264863 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:02:06.264987 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:02:06.265231 19444 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:02:08.070153 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:08.070338 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:12.711354 19483 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:02:12.711539 19483 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:02:12.712275 19483 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:02:12.713088 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:02:12.713256 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:02:12.713382 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:02:12.713614 19483 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:02:13.071692 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:13.071856 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:18.073248 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:18.073431 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:20.735276 19531 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:02:20.735460 19531 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:02:20.736136 19531 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:02:20.737056 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:02:20.738171 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:02:20.738327 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:02:20.738473 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:02:20.738716 19531 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:02:21.735028 19539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:02:21.735229 19539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:02:21.735809 19539 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:02:21.737906 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:02:21.738054 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:02:21.738222 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:02:21.738463 19539 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:02:23.074579 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:23.074757 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:23.500783 19553 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:02:23.501425 19553 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:02:23.502074 19553 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:02:23.503520 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:02:23.503765 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:02:23.503893 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:02:23.504118 19553 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:02:28.075925 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:28.076083 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:29.635048 19587 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:02:29.635219 19587 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:02:29.636057 19587 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:02:29.636956 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:02:29.637122 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:02:29.637220 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:02:29.637424 19587 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:02:33.077404 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:33.077562 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:38.078905 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:38.079082 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:39.427754 19638 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:02:39.427953 19638 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:02:39.429014 19638 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:02:39.429937 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20250111 09:02:39.430936 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:02:39.431069 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:02:39.431200 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:02:39.431425 19638 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:02:42.132944 19657 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:02:42.133106 19657 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:02:42.133767 19657 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:02:42.134598 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:02:42.134783 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:02:42.134938 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:02:42.135193 19657 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:02:43.080441 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:43.080636 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 09:02:43.138065 17390 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (415 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:02:43.356014 17117 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (416 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:02:43.356058 17255 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (416 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:02:48.082011 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:48.082190 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:51.514230 19701 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:02:51.514397 19701 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:02:51.515301 19701 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:02:51.516116 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:02:51.516280 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:02:51.516446 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:02:51.516672 19701 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:02:53.083475 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:53.083642 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:02:57.325666 19735 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:02:57.325872 19735 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:02:57.326681 19735 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:02:57.327528 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:02:57.328624 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:02:57.328775 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:02:57.328891 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:02:57.329110 19735 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:02:58.085240 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:02:58.085412 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:03.086584 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:03.086755 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:06.129328 19784 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:03:06.129591 19784 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:03:06.130421 19784 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:03:06.132031 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:03:06.133244 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:03:06.133379 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:03:06.133491 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:03:06.133687 19784 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:03:08.087975 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:08.088143 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:10.078837 19806 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:03:10.079110 19806 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:03:10.080113 19806 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:03:10.081250 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:03:10.081511 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:03:10.081625 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:03:10.081873 19806 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:03:13.089457 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:13.089629 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:18.090986 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:18.091171 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:18.939234 19852 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:03:18.939401 19852 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:03:18.940495 19852 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:03:18.940829 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:03:18.940991 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:03:18.941121 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:03:18.941332 19852 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:03:21.978624 19867 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:03:21.978811 19867 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:03:21.979676 19867 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:03:21.980552 18687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:03:21.981998 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:03:21.982159 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:03:21.982268 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:03:21.982529 19867 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:03:23.092388 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:23.092571 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:28.093724 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:28.093899 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:29.061321 19908 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:03:29.061542 19908 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:03:29.062168 19908 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:03:29.063055 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:03:29.064291 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:03:29.064466 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:03:29.064579 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:03:29.064774 19908 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:03:33.095180 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:33.095345 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:38.096678 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:38.096849 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:38.326591 19955 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:03:38.326787 19955 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:03:38.327600 19955 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:03:38.328411 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:03:38.328629 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:03:38.328747 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:03:38.329008 19955 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:03:39.095906 19960 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:03:39.096098 19960 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:03:39.096791 19960 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:03:39.097718 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:03:39.098745 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
W20250111 09:03:39.098875 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:03:39.099005 18688 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:03:39.099231 19960 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:03:39.938781 19968 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:03:39.938964 19968 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:03:39.939579 19968 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:03:39.941448 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:03:39.941627 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:03:39.941726 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:03:39.941944 19968 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:03:43.098237 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:43.098418 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250111 09:03:43.259438 17390 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (535 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:03:43.485280 17117 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (536 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:03:43.485294 17255 heartbeater.cc:643] Failed to heartbeat to 127.16.20.190:46487 (536 consecutive failures): Network error: Failed to ping master at 127.16.20.190:46487: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
I20250111 09:03:48.099751 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:48.099915 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:49.558751 20015 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:03:49.558928 20015 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:03:49.560016 20015 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:03:49.560964 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:03:49.561125 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:03:49.561233 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:03:49.561445 20015 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:03:53.101325 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:53.101498 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:54.858109 20044 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:03:54.858330 20044 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:03:54.859197 20044 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:03:54.860082 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250111 09:03:54.861428 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:03:54.861603 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:03:54.861703 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:03:54.861934 20044 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:03:58.102921 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:03:58.103103 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:03:59.460242 20070 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:03:59.460465 20070 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:03:59.461293 20070 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:03:59.462186 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:03:59.462335 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:03:59.462437 17279 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:03:59.462656 20070 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:04:02.686131 20084 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c77238c1f3c9423caf1235b18b67ab2e)
I20250111 09:04:02.686326 20084 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77238c1f3c9423caf1235b18b67ab2e" member_type: VOTER last_known_addr { host: "127.16.20.190" port: 46487 } } peers { permanent_uuid: "77aa4f8aaf704f1898426e17c6fbe28f" member_type: VOTER last_known_addr { host: "127.16.20.189" port: 35063 } } peers { permanent_uuid: "4acb4ec5faa3472f9cecb23f9b5d2e0f" member_type: VOTER last_known_addr { host: "127.16.20.188" port: 41785 } }
I20250111 09:04:02.687009 20084 leader_election.cc:290] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487), 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063)
W20250111 09:04:02.687826 18688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 09:04:02.689170 18688 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer c77238c1f3c9423caf1235b18b67ab2e (127.16.20.190:46487): Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:04:02.689332 18687 leader_election.cc:336] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 77aa4f8aaf704f1898426e17c6fbe28f (127.16.20.189:35063): Network error: Client connection negotiation failed: client connection to 127.16.20.189:35063: connect: Connection refused (error 111)
I20250111 09:04:02.689442 18687 leader_election.cc:304] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4acb4ec5faa3472f9cecb23f9b5d2e0f; no voters: 77aa4f8aaf704f1898426e17c6fbe28f, c77238c1f3c9423caf1235b18b67ab2e
I20250111 09:04:02.689689 20084 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 4acb4ec5faa3472f9cecb23f9b5d2e0f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250111 09:04:03.104137 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:04:03.104319 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:04:05.175578 20100 raft_consensus.cc:491] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 09:04:05.175753 20100 raft_consensus.cc:513] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13afd9f6a62a4efd831797a347bb31b2" member_type: VOTER last_known_addr { host: "127.16.20.129" port: 37699 } } peers { permanent_uuid: "d650c29a4cae41a490fd9aad87215be7" member_type: VOTER last_known_addr { host: "127.16.20.130" port: 35105 } } peers { permanent_uuid: "25e23fc83774472986af5c3f81f591fe" member_type: VOTER last_known_addr { host: "127.16.20.131" port: 33627 } }
I20250111 09:04:05.176419 20100 leader_election.cc:290] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699), d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105)
W20250111 09:04:05.177183 17279 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13afd9f6a62a4efd831797a347bb31b2 (127.16.20.129:37699): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
W20250111 09:04:05.177346 17280 leader_election.cc:343] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d650c29a4cae41a490fd9aad87215be7 (127.16.20.130:35105): Not found: Tablet not found: a2f683bacb4e4778b7ef52d75fdb0832
I20250111 09:04:05.177486 17280 leader_election.cc:304] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25e23fc83774472986af5c3f81f591fe; no voters: 13afd9f6a62a4efd831797a347bb31b2, d650c29a4cae41a490fd9aad87215be7
I20250111 09:04:05.177750 20100 raft_consensus.cc:2743] T a2f683bacb4e4778b7ef52d75fdb0832 P 25e23fc83774472986af5c3f81f591fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 09:04:08.105917 18666 master_runner.cc:186] Error getting master registration for 127.16.20.190:46487: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111), 
I20250111 09:04:08.106078 18666 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250111 09:04:10.550655 18691 maintenance_manager.cc:418] P 4acb4ec5faa3472f9cecb23f9b5d2e0f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1406 bytes on disk
I20250111 09:04:10.555313 18690 maintenance_manager.cc:642] P 4acb4ec5faa3472f9cecb23f9b5d2e0f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s	user 0.003s	sys 0.001s Metrics: {"dirs.queue_time_us":331,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":99,"thread_start_us":410,"threads_started":2,"wal-append.queue_time_us":316}
W20250111 09:04:11.162186 17399 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:04:11.162437 17399 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
W20250111 09:04:11.162545 17399 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
F20250111 09:04:11.173346 17412 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_da4abbdb96ec46e1a5a37a5845eeb580 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1736586251 (unix time) try "date -d @1736586251" if you are using GNU date ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @                0x0 (unknown)
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800004052) received by PID 16466 (TID 0x7fecae3ad700) from PID 16466; stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179353 17405 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_413c709bdd944e2d98cb0e89f0a029ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179353 17405 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_413c709bdd944e2d98cb0e89f0a029ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179430 17406 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_047f7156c794468f8b656960fc0edb9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179353 17405 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_413c709bdd944e2d98cb0e89f0a029ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179430 17406 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_047f7156c794468f8b656960fc0edb9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179641 17414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f0c7f94440643c792a9eed3ea2857f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
    @     0x7fecc4475980 (unknown) at ??:0
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179353 17405 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_413c709bdd944e2d98cb0e89f0a029ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179430 17406 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_047f7156c794468f8b656960fc0edb9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179641 17414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f0c7f94440643c792a9eed3ea2857f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.180907 17408 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a0ad18d08e114a01ac92b944b0d9efec on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
    @     0x7fecbd485fb7 gsignal at ??:0
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179353 17405 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_413c709bdd944e2d98cb0e89f0a029ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179430 17406 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_047f7156c794468f8b656960fc0edb9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179641 17414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f0c7f94440643c792a9eed3ea2857f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.180907 17408 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a0ad18d08e114a01ac92b944b0d9efec on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.181003 17411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5f3d49b148aa4b76b52aff24d64f1ad6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
    @     0x7fecbd487921 abort at ??:0
W20250111 09:04:11.182138 17400 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.16.20.190:46487)
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179353 17405 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_413c709bdd944e2d98cb0e89f0a029ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179430 17406 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_047f7156c794468f8b656960fc0edb9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179641 17414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f0c7f94440643c792a9eed3ea2857f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.180907 17408 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a0ad18d08e114a01ac92b944b0d9efec on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.181003 17411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5f3d49b148aa4b76b52aff24d64f1ad6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.182614 17415 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.182628 17410 master-stress-test.cc:297] Check failed: _s.ok() Bad status: T*** Check failure stack trace: ***
F20250111 09:04:11.173413 17435 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17417 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.173434 17418 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.174688 17409 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1c85c3b17bfe4badafe08b8a5b0ce91b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.175786 17427 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178077 17413 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_518dba2592a54eddbd2b2caec0bb4cfa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178097 17434 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.178153 17407 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7d1113f20be344b0802727c6883a0652 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179353 17405 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_413c709bdd944e2d98cb0e89f0a029ea on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179430 17406 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_047f7156c794468f8b656960fc0edb9b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.179641 17414 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f0c7f94440643c792a9eed3ea2857f0 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.180907 17408 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a0ad18d08e114a01ac92b944b0d9efec on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.181003 17411 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5f3d49b148aa4b76b52aff24d64f1ad6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.182614 17415 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)F20250111 09:04:11.182628 17410 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_137997c870c04b4e95e9ceb8106ce570 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.16.20.190:46487,127.16.20.189:35063,127.16.20.188:41785, num_attempts: 766) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.16.20.190:46487: connect: Connection refused (error 111)
*** Check failure stack trace: ***
    @     0x7fecc01aedcd google::LogMessage::Fail() at ??:0
    @     0x7fecc01b2b93 google::LogMessage::SendToLog() at ??:0
    @     0x7fecc01ae7cc google::LogMessage::Flush() at ??:0
    @     0x7fecc01aff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
    @           0x4085b2 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:297
    @           0x447a84 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
    @           0x44793d std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
    @           0x4478fa _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
    @           0x4478ae std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
    @           0x4475e9 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
    @     0x7fecbdb0d6df (unknown) at ??:0
    @     0x7fecc446a6db start_thread at ??:0
    @     0x7fecbd56871f clone at ??:0