Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-06-16T01:57:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-06-16T01:57:27Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskkSI17J/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskkSI17J/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/security.properties
2026-06-16 01:57:34: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskkSI17J/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskkSI17J/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260616 01:57:42.523499 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.62:35519
--webserver_interface=127.3.56.62
--webserver_port=0
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260616 01:57:42.829556 3580 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:57:42.830165 3580 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:57:42.830603 3580 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260616 01:57:42.862918 3580 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260616 01:57:42.863301 3580 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:57:42.863583 3580 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260616 01:57:42.863854 3580 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260616 01:57:42.901103 3580 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.62:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.56.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:57:42.902622 3580 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:57:42.904300 3580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260616 01:57:42.915251 3586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:44.319512 3585 debug-util.cc:398] Leaking SignalData structure 0x7b08000376c0 after lost signal to thread 3580
W20260616 01:57:44.701321 3585 kernel_stack_watchdog.cc:198] Thread 3580 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260616 01:57:42.916571 3587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:44.702206 3580 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.787s user 0.002s sys 0.000s
W20260616 01:57:44.702567 3580 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.787s user 0.002s sys 0.000s
W20260616 01:57:44.702744 3588 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1785 milliseconds
W20260616 01:57:44.704280 3589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:44.704202 3580 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260616 01:57:44.707494 3580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:57:44.716131 3580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:57:44.717525 3580 hybrid_clock.cc:648] HybridClock initialized: now 1781575064717492 us; error 32 us; skew 500 ppm
I20260616 01:57:44.718356 3580 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:57:44.724191 3580 webserver.cc:533] Webserver started at http://127.3.56.62:37277/ using document root <none> and password file <none>
I20260616 01:57:44.725113 3580 fs_manager.cc:362] Metadata directory not provided
I20260616 01:57:44.725414 3580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:57:44.725965 3580 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260616 01:57:44.730455 3580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data/instance:
uuid: "f509c7b2b84e4bd1a6e09d901dd331b7"
format_stamp: "Formatted at 2026-06-16 01:57:44 on dist-test-slave-s7k7"
I20260616 01:57:44.731568 3580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/wal/instance:
uuid: "f509c7b2b84e4bd1a6e09d901dd331b7"
format_stamp: "Formatted at 2026-06-16 01:57:44 on dist-test-slave-s7k7"
I20260616 01:57:44.738243 3580 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260616 01:57:44.743597 3596 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:44.744592 3580 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260616 01:57:44.744918 3580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/wal
uuid: "f509c7b2b84e4bd1a6e09d901dd331b7"
format_stamp: "Formatted at 2026-06-16 01:57:44 on dist-test-slave-s7k7"
I20260616 01:57:44.745292 3580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:57:44.812644 3580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:57:44.814198 3580 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:57:44.814654 3580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:57:44.883737 3580 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.62:35519
I20260616 01:57:44.883824 3647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.62:35519 every 8 connection(s)
I20260616 01:57:44.886365 3580 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/data/info.pb
I20260616 01:57:44.889803 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 3580
I20260616 01:57:44.890322 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-0/wal/instance
I20260616 01:57:44.893097 3648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:44.895848 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.61:35255
--webserver_interface=127.3.56.61
--webserver_port=0
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260616 01:57:44.906595 3648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
I20260616 01:57:44.935648 3648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } has no permanent_uuid. Determining permanent_uuid...
W20260616 01:57:44.938242 3600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:57:44.942116 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:57:44.991585 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } attempt: 1
W20260616 01:57:44.995848 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:57:45.064242 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } attempt: 2
W20260616 01:57:45.068307 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:57:45.159762 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } attempt: 3
W20260616 01:57:45.163944 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:57:45.204178 3649 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:57:45.204782 3649 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:57:45.205243 3649 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260616 01:57:45.241431 3649 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260616 01:57:45.241888 3649 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:57:45.242277 3649 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260616 01:57:45.242516 3649 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260616 01:57:45.280488 3649 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.61:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.56.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:57:45.281996 3649 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:57:45.283622 3649 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260616 01:57:45.294920 3658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:45.307308 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } attempt: 4
W20260616 01:57:45.314195 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:57:45.613813 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } attempt: 5
W20260616 01:57:45.620868 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:57:46.168502 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } attempt: 6
W20260616 01:57:46.179296 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:57:45.296469 3659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:45.299832 3661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:46.513651 3660 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1214 milliseconds
I20260616 01:57:46.513769 3649 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260616 01:57:46.514997 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:57:46.518005 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:57:46.519445 3649 hybrid_clock.cc:648] HybridClock initialized: now 1781575066519414 us; error 48 us; skew 500 ppm
I20260616 01:57:46.520183 3649 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:57:46.526124 3649 webserver.cc:533] Webserver started at http://127.3.56.61:36479/ using document root <none> and password file <none>
I20260616 01:57:46.526959 3649 fs_manager.cc:362] Metadata directory not provided
I20260616 01:57:46.527132 3649 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:57:46.527529 3649 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260616 01:57:46.531692 3649 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data/instance:
uuid: "321530d69d794e8f94c5424224a44115"
format_stamp: "Formatted at 2026-06-16 01:57:46 on dist-test-slave-s7k7"
I20260616 01:57:46.532698 3649 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/wal/instance:
uuid: "321530d69d794e8f94c5424224a44115"
format_stamp: "Formatted at 2026-06-16 01:57:46 on dist-test-slave-s7k7"
I20260616 01:57:46.539477 3649 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260616 01:57:46.544806 3670 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:46.545866 3649 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260616 01:57:46.546183 3649 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/wal
uuid: "321530d69d794e8f94c5424224a44115"
format_stamp: "Formatted at 2026-06-16 01:57:46 on dist-test-slave-s7k7"
I20260616 01:57:46.546563 3649 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:57:46.599802 3649 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:57:46.601285 3649 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:57:46.601727 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:57:46.670045 3649 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.61:35255
I20260616 01:57:46.670120 3721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.61:35255 every 8 connection(s)
I20260616 01:57:46.672722 3649 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/data/info.pb
I20260616 01:57:46.678241 3722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:46.680673 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 3649
I20260616 01:57:46.681167 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-1/wal/instance
I20260616 01:57:46.683341 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.60:37239
--webserver_interface=127.3.56.60
--webserver_port=0
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260616 01:57:46.688872 3722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
I20260616 01:57:46.709286 3722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } has no permanent_uuid. Determining permanent_uuid...
I20260616 01:57:46.723978 3722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
W20260616 01:57:46.726852 3673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
W20260616 01:57:46.731736 3722 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:46.781306 3722 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 1
W20260616 01:57:46.785701 3722 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:46.854102 3722 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 2
W20260616 01:57:46.858435 3722 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:46.949838 3722 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 3
W20260616 01:57:46.953965 3722 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
W20260616 01:57:47.002275 3723 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:57:47.002885 3723 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:57:47.003306 3723 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260616 01:57:47.035917 3723 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260616 01:57:47.036257 3723 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:57:47.036532 3723 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260616 01:57:47.036792 3723 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260616 01:57:47.074270 3723 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.60:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.56.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:57:47.075600 3723 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:57:47.077284 3723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260616 01:57:47.087374 3733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:47.097312 3722 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 4
W20260616 01:57:47.104298 3722 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.239936 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } attempt: 7
I20260616 01:57:47.283846 3648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
W20260616 01:57:47.303891 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.362445 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 1
W20260616 01:57:47.376406 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.403824 3722 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 5
W20260616 01:57:47.412504 3722 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.458027 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 2
W20260616 01:57:47.466363 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.552800 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 3
W20260616 01:57:47.561105 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.701764 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 4
W20260616 01:57:47.711812 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.960095 3722 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 6
W20260616 01:57:47.982470 3722 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:47.995368 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 5
W20260616 01:57:48.003484 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
W20260616 01:57:47.088991 3734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:48.403657 3735 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1313 milliseconds
W20260616 01:57:48.404232 3736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:48.407413 3723 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.319s user 0.490s sys 0.819s
W20260616 01:57:48.407716 3723 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.319s user 0.490s sys 0.819s
I20260616 01:57:48.407939 3723 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260616 01:57:48.409021 3723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:57:48.411499 3723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:57:48.412930 3723 hybrid_clock.cc:648] HybridClock initialized: now 1781575068412880 us; error 38 us; skew 500 ppm
I20260616 01:57:48.413826 3723 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:57:48.421464 3723 webserver.cc:533] Webserver started at http://127.3.56.60:34221/ using document root <none> and password file <none>
I20260616 01:57:48.422463 3723 fs_manager.cc:362] Metadata directory not provided
I20260616 01:57:48.422701 3723 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:57:48.423151 3723 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260616 01:57:48.427441 3723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/instance:
uuid: "673bf665c7d743579440c02560de26b9"
format_stamp: "Formatted at 2026-06-16 01:57:48 on dist-test-slave-s7k7"
I20260616 01:57:48.428542 3723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal/instance:
uuid: "673bf665c7d743579440c02560de26b9"
format_stamp: "Formatted at 2026-06-16 01:57:48 on dist-test-slave-s7k7"
I20260616 01:57:48.436343 3723 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260616 01:57:48.442409 3749 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:48.443564 3723 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260616 01:57:48.443876 3723 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
uuid: "673bf665c7d743579440c02560de26b9"
format_stamp: "Formatted at 2026-06-16 01:57:48 on dist-test-slave-s7k7"
I20260616 01:57:48.444206 3723 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:57:48.512684 3723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:57:48.514456 3723 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:57:48.514891 3723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:57:48.556020 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 6
W20260616 01:57:48.564431 3648 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.3.56.60:37239: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:48.584872 3723 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.60:37239
I20260616 01:57:48.584965 3802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.60:37239 every 8 connection(s)
I20260616 01:57:48.587541 3723 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
I20260616 01:57:48.592893 3803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:48.596869 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 3723
I20260616 01:57:48.597522 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal/instance
I20260616 01:57:48.607407 3803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
I20260616 01:57:48.628556 3803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } has no permanent_uuid. Determining permanent_uuid...
I20260616 01:57:48.639721 3803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
I20260616 01:57:48.658986 3803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Bootstrap starting.
I20260616 01:57:48.665856 3803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:48.668123 3803 log.cc:826] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Log is configured to *not* fsync() on all Append() calls
I20260616 01:57:48.672824 3803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: No bootstrap required, opened a new log
I20260616 01:57:48.693157 3803 raft_consensus.cc:359] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:48.694053 3803 raft_consensus.cc:385] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:48.694311 3803 raft_consensus.cc:740] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673bf665c7d743579440c02560de26b9, State: Initialized, Role: FOLLOWER
I20260616 01:57:48.695013 3803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:48.697182 3813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:48.697780 3813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:48.698657 3803 sys_catalog.cc:565] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: configured and running, proceeding with master startup.
W20260616 01:57:48.732667 3827 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260616 01:57:48.733105 3827 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260616 01:57:48.724972 3723 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: OK, instance_id { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" instance_seqno: 1781575064864755 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260616 01:57:48.756836 3723 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 01:57:48.770627 3755 tablet.cc:2406] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260616 01:57:48.992304 3813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:57:48.992879 3813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:48.995441 3813 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
I20260616 01:57:49.043040 3722 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 7
I20260616 01:57:49.069037 3722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: Bootstrap starting.
I20260616 01:57:49.076772 3722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:49.079314 3722 log.cc:826] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: Log is configured to *not* fsync() on all Append() calls
I20260616 01:57:49.088018 3722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: No bootstrap required, opened a new log
I20260616 01:57:49.116679 3722 raft_consensus.cc:359] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:49.117700 3722 raft_consensus.cc:385] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:49.118039 3722 raft_consensus.cc:740] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321530d69d794e8f94c5424224a44115, State: Initialized, Role: FOLLOWER
I20260616 01:57:49.119010 3722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:49.123558 3832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.125272 3722 sys_catalog.cc:565] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: configured and running, proceeding with master startup.
I20260616 01:57:49.125370 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673bf665c7d743579440c02560de26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:49.126268 3697 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673bf665c7d743579440c02560de26b9 in term 0.
I20260616 01:57:49.128247 3753 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9; no voters:
I20260616 01:57:49.129431 3813 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:57:49.129848 3813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:57:49.130198 3813 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:49.138450 3813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:49.124908 3832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:49.144088 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673bf665c7d743579440c02560de26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115"
I20260616 01:57:49.144806 3697 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:49.162739 3697 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673bf665c7d743579440c02560de26b9 in term 1.
I20260616 01:57:49.163384 3813 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 1 election: Requested vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
I20260616 01:57:49.165005 3753 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9; no voters:
I20260616 01:57:49.166075 3813 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260616 01:57:49.168270 3813 raft_consensus.cc:697] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 1 LEADER]: Becoming Leader. State: Replica: 673bf665c7d743579440c02560de26b9, State: Running, Role: LEADER
I20260616 01:57:49.169613 3813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:49.179690 3840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 673bf665c7d743579440c02560de26b9. Latest consensus state: current_term: 1 leader_uuid: "673bf665c7d743579440c02560de26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.180431 3840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: LEADER
I20260616 01:57:49.190303 3843 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260616 01:57:49.200604 3843 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260616 01:57:49.239661 3697 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Refusing update from remote peer 673bf665c7d743579440c02560de26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260616 01:57:49.243203 3813 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260616 01:57:49.249985 3851 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260616 01:57:49.263216 3832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 673bf665c7d743579440c02560de26b9. Latest consensus state: current_term: 1 leader_uuid: "673bf665c7d743579440c02560de26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.263906 3832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:49.274487 3813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "673bf665c7d743579440c02560de26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.275229 3813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: LEADER
I20260616 01:57:49.283165 3843 catalog_manager.cc:1382] Generated new cluster ID: 640ff0fe2fa14eee9f5b81f68327cf28
I20260616 01:57:49.283459 3843 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260616 01:57:49.308101 3832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "673bf665c7d743579440c02560de26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.308818 3832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:49.255859 3649 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: OK, instance_id { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" instance_seqno: 1781575064864755 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260616 01:57:49.310493 3649 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:57:49.341778 3843 catalog_manager.cc:1405] Generated new certificate authority record
I20260616 01:57:49.343737 3843 catalog_manager.cc:1539] Loading token signing keys...
I20260616 01:57:49.363651 3843 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Generated new TSK 0
I20260616 01:57:49.364653 3843 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260616 01:57:49.369127 3843 catalog_manager.cc:1562] Loading latest processed Hive Metastore notification log event ID...
I20260616 01:57:49.373363 3843 catalog_manager.cc:5918] Last processed Hive Metastore notification event ID: -1
I20260616 01:57:49.390534 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.56.1:0
--local_ip_for_outbound_sockets=127.3.56.1
--webserver_interface=127.3.56.1
--webserver_port=0
--tserver_master_addrs=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260616 01:57:49.597854 3648 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } attempt: 7
I20260616 01:57:49.644786 3648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Bootstrap starting.
I20260616 01:57:49.656561 3648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:49.662372 3648 log.cc:826] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Log is configured to *not* fsync() on all Append() calls
I20260616 01:57:49.685050 3648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: No bootstrap required, opened a new log
W20260616 01:57:49.696523 3751 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.56.62:35519 timed out after 0.554s (SENT)
I20260616 01:57:49.774430 3648 raft_consensus.cc:359] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:49.783088 3648 raft_consensus.cc:385] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:49.792131 3648 raft_consensus.cc:740] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f509c7b2b84e4bd1a6e09d901dd331b7, State: Initialized, Role: FOLLOWER
I20260616 01:57:49.837844 3648 consensus_queue.cc:260] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:49.842996 3648 sys_catalog.cc:565] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260616 01:57:49.843518 3621 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:49.845470 3622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673bf665c7d743579440c02560de26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7"
W20260616 01:57:49.846525 3622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42226 (request call id 3) took 704 ms (client timeout 554 ms). Trace:
W20260616 01:57:49.846858 3622 rpcz_store.cc:269] 0616 01:57:49.142108 (+ 0us) service_pool.cc:167] Inserting onto call queue
0616 01:57:49.142243 (+ 135us) service_pool.cc:224] Handling call
0616 01:57:49.846442 (+704199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260616 01:57:49.847997 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673bf665c7d743579440c02560de26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
W20260616 01:57:49.848783 3623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42226 (request call id 2) took 852 ms (client timeout 745 ms). Trace:
W20260616 01:57:49.849086 3623 rpcz_store.cc:269] 0616 01:57:48.996146 (+ 0us) service_pool.cc:167] Inserting onto call queue
0616 01:57:48.996296 (+ 150us) service_pool.cc:224] Handling call
0616 01:57:49.848725 (+852429us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260616 01:57:49.849973 3619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673bf665c7d743579440c02560de26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:49.851046 3620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "673bf665c7d743579440c02560de26b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7"
I20260616 01:57:49.859861 3621 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Refusing update from remote peer 673bf665c7d743579440c02560de26b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260616 01:57:49.862011 3813 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260616 01:57:49.869814 3865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.871635 3865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:49.903223 3871 mvcc.cc:204] Tried to move back new op lower bound from 7297331483562516480 to 7297331483332546560. Current Snapshot: MvccSnapshot[applied={T|T < 7297331483562516480}]
W20260616 01:57:49.910092 3856 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:57:49.910936 3856 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:57:49.911958 3856 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260616 01:57:49.930040 3869 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "673bf665c7d743579440c02560de26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.930837 3869 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:49.928920 3865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 673bf665c7d743579440c02560de26b9. Latest consensus state: current_term: 1 leader_uuid: "673bf665c7d743579440c02560de26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:49.931878 3865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: This master's current role is: FOLLOWER
W20260616 01:57:49.977035 3856 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:57:49.978582 3856 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.56.1
I20260616 01:57:49.987154 3885 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260616 01:57:50.037664 3856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.56.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.56.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.56.1
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:57:50.038990 3856 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:57:50.040613 3856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260616 01:57:50.052754 3887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:50.274358 3851 catalog_manager.cc:1294] Loaded cluster ID: 640ff0fe2fa14eee9f5b81f68327cf28
I20260616 01:57:50.274750 3851 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: loading cluster ID for follower catalog manager: success
I20260616 01:57:50.295996 3851 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: acquiring CA information for follower catalog manager: success
I20260616 01:57:50.309909 3851 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260616 01:57:50.380534 3865 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:50.381124 3865 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:50.383055 3813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "673bf665c7d743579440c02560de26b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:50.384047 3865 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:50.384678 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:50.385504 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:50.388285 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:50.389088 3865 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:50.383991 3813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: LEADER
I20260616 01:57:50.692912 3897 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:50.693650 3897 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:50.697229 3897 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:50.697232 3620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:50.707901 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:50.709534 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:50.710851 3897 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:51.011925 3885 catalog_manager.cc:1294] Loaded cluster ID: 640ff0fe2fa14eee9f5b81f68327cf28
I20260616 01:57:51.017295 3885 catalog_manager.cc:1587] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: loading cluster ID for follower catalog manager: success
I20260616 01:57:51.050127 3885 catalog_manager.cc:1609] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: acquiring CA information for follower catalog manager: success
I20260616 01:57:51.056041 3885 catalog_manager.cc:1637] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260616 01:57:51.249522 3899 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:51.251761 3899 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:51.256417 3620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:51.262070 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:51.264358 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:51.277976 3900 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:51.278852 3899 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:51.852067 3901 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:51.854195 3901 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:51.858697 3620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:51.860924 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:51.862433 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:51.865707 3901 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:51.866317 3901 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:52.104979 3903 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:52.113514 3903 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:52.118760 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:52.123283 3903 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:52.124446 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:52.126096 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:52.129370 3903 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260616 01:57:51.456446 3886 debug-util.cc:398] Leaking SignalData structure 0x7b080002eea0 after lost signal to thread 3856
W20260616 01:57:52.389025 3889 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2333 milliseconds
W20260616 01:57:52.389308 3856 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.334s user 0.000s sys 0.002s
W20260616 01:57:52.390372 3856 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.336s user 0.000s sys 0.002s
W20260616 01:57:50.053867 3888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:52.395714 3856 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260616 01:57:52.396252 3891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:52.397697 3856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:57:52.408496 3856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:57:52.412333 3856 hybrid_clock.cc:648] HybridClock initialized: now 1781575072412231 us; error 87 us; skew 500 ppm
I20260616 01:57:52.413524 3856 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:57:52.446347 3907 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:52.447665 3856 webserver.cc:533] Webserver started at http://127.3.56.1:35973/ using document root <none> and password file <none>
I20260616 01:57:52.448999 3856 fs_manager.cc:362] Metadata directory not provided
I20260616 01:57:52.447499 3907 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:52.449313 3856 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:57:52.449911 3856 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260616 01:57:52.451812 3620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:52.460457 3856 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data/instance:
uuid: "dc87c967de1144a6986d9ff87150d21d"
format_stamp: "Formatted at 2026-06-16 01:57:52 on dist-test-slave-s7k7"
I20260616 01:57:52.461992 3856 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal/instance:
uuid: "dc87c967de1144a6986d9ff87150d21d"
format_stamp: "Formatted at 2026-06-16 01:57:52 on dist-test-slave-s7k7"
I20260616 01:57:52.462344 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:52.463932 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:52.466600 3907 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:52.467413 3907 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:52.478222 3856 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.014s sys 0.001s
I20260616 01:57:52.491627 3910 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:52.492903 3856 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260616 01:57:52.493336 3856 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal
uuid: "dc87c967de1144a6986d9ff87150d21d"
format_stamp: "Formatted at 2026-06-16 01:57:52 on dist-test-slave-s7k7"
I20260616 01:57:52.494065 3856 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:57:52.618449 3856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:57:52.621130 3856 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:57:52.621735 3856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:57:52.637403 3856 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260616 01:57:52.649468 3856 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260616 01:57:52.649760 3856 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:52.650079 3856 ts_tablet_manager.cc:616] Registered 0 tablets
I20260616 01:57:52.650429 3856 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:52.990983 3856 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.1:34353
I20260616 01:57:52.992079 4023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.1:34353 every 8 connection(s)
I20260616 01:57:52.995627 3856 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/data/info.pb
I20260616 01:57:53.003069 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 3856
I20260616 01:57:53.003556 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal/instance
I20260616 01:57:53.051649 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.56.2:0
--local_ip_for_outbound_sockets=127.3.56.2
--webserver_interface=127.3.56.2
--webserver_port=0
--tserver_master_addrs=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260616 01:57:53.074369 4025 heartbeater.cc:344] Connected to a master server at 127.3.56.61:35255
I20260616 01:57:53.074864 4025 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:53.076211 4025 heartbeater.cc:507] Master 127.3.56.61:35255 requested a full tablet report, sending...
I20260616 01:57:53.079766 3687 ts_manager.cc:194] Registered new tserver with Master: dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353)
I20260616 01:57:53.093863 4024 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:57:53.094206 4024 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:53.095129 4024 heartbeater.cc:507] Master 127.3.56.60:37239 requested a full tablet report, sending...
I20260616 01:57:53.098697 3766 ts_manager.cc:194] Registered new tserver with Master: dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353)
I20260616 01:57:53.101665 3766 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.56.1:37559
I20260616 01:57:53.109752 4030 heartbeater.cc:344] Connected to a master server at 127.3.56.62:35519
I20260616 01:57:53.110118 4030 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:53.110785 4030 heartbeater.cc:507] Master 127.3.56.62:35519 requested a full tablet report, sending...
I20260616 01:57:53.114149 3613 ts_manager.cc:194] Registered new tserver with Master: dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353)
I20260616 01:57:53.141686 4035 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:53.142485 4035 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:53.145911 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:53.148947 4035 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:53.149139 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:53.150728 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:53.151808 4035 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260616 01:57:53.450022 4034 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:57:53.450554 4034 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:57:53.451128 4034 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260616 01:57:53.485062 4034 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:57:53.486047 4034 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.56.2
I20260616 01:57:53.524686 4034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.56.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.56.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.56.2
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:57:53.526115 4034 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:57:53.527866 4034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260616 01:57:53.553880 4041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:53.606031 4024 heartbeater.cc:499] Master 127.3.56.60:37239 was elected leader, sending a full tablet report...
I20260616 01:57:53.871809 4047 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:53.872751 4047 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:53.877342 3620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:53.880402 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:53.882547 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:53.894462 4047 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:53.895901 4047 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:54.241868 4049 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:54.242725 4049 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:54.246241 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:54.250720 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:54.252785 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:54.255344 4049 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:54.256407 4049 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:54.846058 4051 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:54.846630 4051 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:54.851814 4051 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:54.852466 3621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:54.855898 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:54.857870 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:54.858807 4051 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260616 01:57:54.964011 4040 debug-util.cc:398] Leaking SignalData structure 0x7b080002eea0 after lost signal to thread 4034
W20260616 01:57:53.557643 4042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:55.219518 4040 kernel_stack_watchdog.cc:198] Thread 4034 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260616 01:57:55.221004 4034 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.665s user 0.396s sys 0.799s
W20260616 01:57:55.221488 4034 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.666s user 0.396s sys 0.799s
W20260616 01:57:55.222596 4043 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1665 milliseconds
W20260616 01:57:55.222913 4044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:55.223022 4034 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260616 01:57:55.231673 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:57:55.234427 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:57:55.235888 4034 hybrid_clock.cc:648] HybridClock initialized: now 1781575075235850 us; error 51 us; skew 500 ppm
I20260616 01:57:55.237053 4034 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:57:55.244854 4034 webserver.cc:533] Webserver started at http://127.3.56.2:34505/ using document root <none> and password file <none>
I20260616 01:57:55.246162 4034 fs_manager.cc:362] Metadata directory not provided
I20260616 01:57:55.246457 4034 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:57:55.247005 4034 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260616 01:57:55.253146 4034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data/instance:
uuid: "f4d208b546004421a8157a05464cbe7e"
format_stamp: "Formatted at 2026-06-16 01:57:55 on dist-test-slave-s7k7"
I20260616 01:57:55.254616 4034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal/instance:
uuid: "f4d208b546004421a8157a05464cbe7e"
format_stamp: "Formatted at 2026-06-16 01:57:55 on dist-test-slave-s7k7"
I20260616 01:57:55.263051 4034 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260616 01:57:55.268424 4057 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:55.269520 4034 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260616 01:57:55.269837 4034 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal
uuid: "f4d208b546004421a8157a05464cbe7e"
format_stamp: "Formatted at 2026-06-16 01:57:55 on dist-test-slave-s7k7"
I20260616 01:57:55.270190 4034 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:57:55.343622 4034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:57:55.345203 4034 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:57:55.345685 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:57:55.348235 4034 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260616 01:57:55.352308 4034 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260616 01:57:55.352536 4034 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:55.352780 4034 ts_tablet_manager.cc:616] Registered 0 tablets
I20260616 01:57:55.352953 4034 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:55.413502 4104 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:55.414350 4104 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:55.417081 4104 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:55.417579 3621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:55.418711 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:55.420212 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:55.421185 4104 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:55.493017 4034 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.2:41579
I20260616 01:57:55.493176 4170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.2:41579 every 8 connection(s)
I20260616 01:57:55.496234 4034 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/data/info.pb
I20260616 01:57:55.496961 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 4034
I20260616 01:57:55.497399 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal/instance
I20260616 01:57:55.532039 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.56.3:0
--local_ip_for_outbound_sockets=127.3.56.3
--webserver_interface=127.3.56.3
--webserver_port=0
--tserver_master_addrs=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260616 01:57:55.549259 4172 heartbeater.cc:344] Connected to a master server at 127.3.56.61:35255
I20260616 01:57:55.549593 4173 heartbeater.cc:344] Connected to a master server at 127.3.56.62:35519
I20260616 01:57:55.549751 4172 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:55.549968 4173 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:55.551306 4173 heartbeater.cc:507] Master 127.3.56.62:35519 requested a full tablet report, sending...
I20260616 01:57:55.551359 4172 heartbeater.cc:507] Master 127.3.56.61:35255 requested a full tablet report, sending...
I20260616 01:57:55.552417 4171 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:57:55.552788 4171 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:55.553766 4171 heartbeater.cc:507] Master 127.3.56.60:37239 requested a full tablet report, sending...
I20260616 01:57:55.554334 3613 ts_manager.cc:194] Registered new tserver with Master: f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:57:55.556030 3766 ts_manager.cc:194] Registered new tserver with Master: f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:57:55.556717 3687 ts_manager.cc:194] Registered new tserver with Master: f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:57:55.557821 3766 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.56.2:51483
W20260616 01:57:55.872152 4181 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:57:55.872856 4181 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:57:55.873498 4181 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260616 01:57:55.930110 4181 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:57:55.931083 4181 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.56.3
I20260616 01:57:55.968479 4181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.56.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.56.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.56.3
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:57:55.969869 4181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:57:55.971468 4181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260616 01:57:55.984848 4187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:57:56.006842 4192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:56.007647 4192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:56.010834 3621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:56.012720 4192 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:56.013751 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:56.015117 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:56.015866 4192 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:56.061228 4171 heartbeater.cc:499] Master 127.3.56.60:37239 was elected leader, sending a full tablet report...
I20260616 01:57:56.609478 4194 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:56.609963 4194 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:56.612278 4194 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:56.612390 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:56.613006 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:56.614449 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:56.615417 4194 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:57.218844 4196 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:57.219312 4196 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:57.221339 4196 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:57.222478 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:57.224357 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:57.225972 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:57.226729 4196 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260616 01:57:55.985781 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:57.263895 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:57:57.266530 4189 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1276 milliseconds
I20260616 01:57:57.266621 4181 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260616 01:57:57.267815 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:57:57.270403 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:57:57.271804 4181 hybrid_clock.cc:648] HybridClock initialized: now 1781575077271759 us; error 47 us; skew 500 ppm
I20260616 01:57:57.272560 4181 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:57:57.278496 4181 webserver.cc:533] Webserver started at http://127.3.56.3:42491/ using document root <none> and password file <none>
I20260616 01:57:57.279376 4181 fs_manager.cc:362] Metadata directory not provided
I20260616 01:57:57.279588 4181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:57:57.280027 4181 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260616 01:57:57.284384 4181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data/instance:
uuid: "3254c8a756fb47a78c049537dbe5e110"
format_stamp: "Formatted at 2026-06-16 01:57:57 on dist-test-slave-s7k7"
I20260616 01:57:57.285501 4181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal/instance:
uuid: "3254c8a756fb47a78c049537dbe5e110"
format_stamp: "Formatted at 2026-06-16 01:57:57 on dist-test-slave-s7k7"
I20260616 01:57:57.292192 4181 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260616 01:57:57.297591 4201 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:57.298566 4181 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260616 01:57:57.298882 4181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal
uuid: "3254c8a756fb47a78c049537dbe5e110"
format_stamp: "Formatted at 2026-06-16 01:57:57 on dist-test-slave-s7k7"
I20260616 01:57:57.299201 4181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:57:57.351015 4181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:57:57.352463 4181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:57:57.352892 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:57:57.355379 4181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260616 01:57:57.359302 4181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260616 01:57:57.359535 4181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:57.359778 4181 ts_tablet_manager.cc:616] Registered 0 tablets
I20260616 01:57:57.359963 4181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260616 01:57:57.429035 4258 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:57.429602 4258 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:57.432533 4258 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:57.432579 3621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:57.433126 3776 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:57:57.434917 3673 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 321530d69d794e8f94c5424224a44115; no voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:57:57.436046 4258 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:57.495885 4181 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.3:41531
I20260616 01:57:57.496050 4314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.3:41531 every 8 connection(s)
I20260616 01:57:57.498431 4181 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/data/info.pb
I20260616 01:57:57.506162 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 4181
I20260616 01:57:57.506656 3296 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal/instance
I20260616 01:57:57.535192 4317 heartbeater.cc:344] Connected to a master server at 127.3.56.62:35519
I20260616 01:57:57.535609 4315 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:57:57.535717 4317 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:57.535945 4315 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:57.536404 4316 heartbeater.cc:344] Connected to a master server at 127.3.56.61:35255
I20260616 01:57:57.536742 4316 heartbeater.cc:461] Registering TS with master...
I20260616 01:57:57.537130 4317 heartbeater.cc:507] Master 127.3.56.62:35519 requested a full tablet report, sending...
I20260616 01:57:57.537127 4315 heartbeater.cc:507] Master 127.3.56.60:37239 requested a full tablet report, sending...
I20260616 01:57:57.537652 4316 heartbeater.cc:507] Master 127.3.56.61:35255 requested a full tablet report, sending...
I20260616 01:57:57.539814 3766 ts_manager.cc:194] Registered new tserver with Master: 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:57.540252 3687 ts_manager.cc:194] Registered new tserver with Master: 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:57.540252 3613 ts_manager.cc:194] Registered new tserver with Master: 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:57.541373 3766 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.3.56.3:60197
I20260616 01:57:57.549860 3296 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260616 01:57:57.605973 3766 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_393cb35288b14df58c97c418d0db05b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:57.610666 3765 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_57c462b8048f489fac3e5529096d3f59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:57.613018 3764 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_59ad6b6ba51b44858e1003fece56722a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:57.615383 3763 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36506:
name: "default.Table_3692a87561694c028a780ad53630b478"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260616 01:57:57.621730 3766 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_393cb35288b14df58c97c418d0db05b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260616 01:57:57.646246 3763 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3692a87561694c028a780ad53630b478 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260616 01:57:57.646565 3765 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_57c462b8048f489fac3e5529096d3f59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260616 01:57:57.650485 3764 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_59ad6b6ba51b44858e1003fece56722a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260616 01:57:57.673683 3296 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu with pid 3723
W20260616 01:57:57.715688 4327 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.56.60:37239)
I20260616 01:57:57.745040 4350 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:57.749467 4350 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:57.752645 3697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "321530d69d794e8f94c5424224a44115" is_pre_election: true
I20260616 01:57:57.760460 4350 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
W20260616 01:57:57.772198 3599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20260616 01:57:57.794064 3599 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239): Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:57.796907 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:57.798156 4350 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260616 01:57:57.989807 4355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:57.990242 4355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:57.999186 4355 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:57.999379 3621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" is_pre_election: true
I20260616 01:57:58.000121 3621 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 321530d69d794e8f94c5424224a44115 in term 1.
I20260616 01:57:58.002213 3672 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7; no voters:
I20260616 01:57:58.003250 4355 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260616 01:57:58.003556 4355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 673bf665c7d743579440c02560de26b9)
I20260616 01:57:58.003861 4355 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 1 FOLLOWER]: Advancing to term 2
I20260616 01:57:58.012051 4355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
W20260616 01:57:58.014058 3673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260616 01:57:58.033147 4355 leader_election.cc:290] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 election: Requested vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:57:58.034235 3621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "321530d69d794e8f94c5424224a44115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7"
I20260616 01:57:58.034914 3621 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 1 FOLLOWER]: Advancing to term 2
I20260616 01:57:58.044584 4315 heartbeater.cc:499] Master 127.3.56.60:37239 was elected leader, sending a full tablet report...
I20260616 01:57:58.051700 3621 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 321530d69d794e8f94c5424224a44115 in term 2.
W20260616 01:57:58.056042 3673 leader_election.cc:336] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239): Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:58.056632 3672 leader_election.cc:304] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7; no voters:
I20260616 01:57:58.057401 4355 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 2 FOLLOWER]: Leader election won for term 2
W20260616 01:57:58.062592 3673 leader_election.cc:336] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239): Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
W20260616 01:57:58.062602 4315 heartbeater.cc:646] Failed to heartbeat to 127.3.56.60:37239 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:58.068842 4355 raft_consensus.cc:697] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [term 2 LEADER]: Becoming Leader. State: Replica: 321530d69d794e8f94c5424224a44115, State: Running, Role: LEADER
I20260616 01:57:58.069968 4355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:58.091254 4359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 321530d69d794e8f94c5424224a44115. Latest consensus state: current_term: 2 leader_uuid: "321530d69d794e8f94c5424224a44115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:58.096750 4359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: This master's current role is: LEADER
W20260616 01:57:58.106211 4171 heartbeater.cc:646] Failed to heartbeat to 127.3.56.60:37239 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:58.122689 4362 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260616 01:57:58.130900 4362 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260616 01:57:58.136035 4362 catalog_manager.cc:1294] Loaded cluster ID: 640ff0fe2fa14eee9f5b81f68327cf28
I20260616 01:57:58.136262 4362 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260616 01:57:58.140969 4362 catalog_manager.cc:1539] Loading token signing keys...
I20260616 01:57:58.147053 4362 catalog_manager.cc:6080] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115: Loaded TSK: 0
I20260616 01:57:58.148530 4362 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260616 01:57:58.155028 4362 catalog_manager.cc:1562] Loading latest processed Hive Metastore notification log event ID...
W20260616 01:57:58.161651 4024 heartbeater.cc:646] Failed to heartbeat to 127.3.56.60:37239 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:58.161257 4362 catalog_manager.cc:5918] Last processed Hive Metastore notification event ID: -1
I20260616 01:57:58.182574 3687 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_3692a87561694c028a780ad53630b478"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:58.185549 3686 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_59ad6b6ba51b44858e1003fece56722a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:58.186360 3685 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_57c462b8048f489fac3e5529096d3f59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:58.189769 3684 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_393cb35288b14df58c97c418d0db05b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260616 01:57:58.196171 3687 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3692a87561694c028a780ad53630b478 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260616 01:57:58.209522 3686 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_59ad6b6ba51b44858e1003fece56722a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260616 01:57:58.213332 3685 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_57c462b8048f489fac3e5529096d3f59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260616 01:57:58.216890 3684 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_393cb35288b14df58c97c418d0db05b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260616 01:57:58.532153 3686 catalog_manager.cc:2199] created HMS catalog entry for table [id=6035dfb0b8d94f3c981a30d83469e645]
I20260616 01:57:58.555474 3621 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Refusing update from remote peer 321530d69d794e8f94c5424224a44115: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260616 01:57:58.557156 4355 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260616 01:57:58.562075 3673 consensus_peers.cc:597] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 -> Peer 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239): Couldn't send request to peer 673bf665c7d743579440c02560de26b9. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260616 01:57:58.600903 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 321530d69d794e8f94c5424224a44115. Latest consensus state: current_term: 2 leader_uuid: "321530d69d794e8f94c5424224a44115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:58.601672 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:58.605738 3685 catalog_manager.cc:2199] created HMS catalog entry for table [id=2ba9c42af02e485bb8990524ac09da17]
I20260616 01:57:58.608192 4364 mvcc.cc:204] Tried to move back new op lower bound from 7297331521718722560 to 7297331519817850880. Current Snapshot: MvccSnapshot[applied={T|T < 7297331521718722560}]
I20260616 01:57:58.626368 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "321530d69d794e8f94c5424224a44115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:58.627167 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:57:58.649593 3686 catalog_manager.cc:3569] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36708:
table { table_name: "default.Table_59ad6b6ba51b44858e1003fece56722a" } new_table_name: "default.Table_100c8bee45e34b27a94e7835f0878e64" modify_external_catalogs: true
I20260616 01:57:58.664103 4359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "321530d69d794e8f94c5424224a44115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:57:58.664876 4359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 321530d69d794e8f94c5424224a44115 [sys.catalog]: This master's current role is: LEADER
I20260616 01:57:58.696933 3685 catalog_manager.cc:3569] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36708:
table { table_name: "default.Table_57c462b8048f489fac3e5529096d3f59" } new_table_name: "default.Table_5afd5bde5bcd4fe3928ccb78ecfa3824" modify_external_catalogs: true
I20260616 01:57:58.744275 3958 tablet_service.cc:1511] Processing CreateTablet for tablet 3a719e057ca84bbb9dcb3a8320777481 (DEFAULT_TABLE table=default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:58.746377 3958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a719e057ca84bbb9dcb3a8320777481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:58.769889 3957 tablet_service.cc:1511] Processing CreateTablet for tablet 864f06b09c484e3a960d149550db97b6 (DEFAULT_TABLE table=default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:58.771186 3957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 864f06b09c484e3a960d149550db97b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:58.807432 4106 tablet_service.cc:1511] Processing CreateTablet for tablet 3a719e057ca84bbb9dcb3a8320777481 (DEFAULT_TABLE table=default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:58.809113 4105 tablet_service.cc:1511] Processing CreateTablet for tablet 864f06b09c484e3a960d149550db97b6 (DEFAULT_TABLE table=default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:58.810622 4105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 864f06b09c484e3a960d149550db97b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:58.811327 4106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a719e057ca84bbb9dcb3a8320777481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:58.841922 4249 tablet_service.cc:1511] Processing CreateTablet for tablet 3a719e057ca84bbb9dcb3a8320777481 (DEFAULT_TABLE table=default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:58.843842 4249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a719e057ca84bbb9dcb3a8320777481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:58.846333 4248 tablet_service.cc:1511] Processing CreateTablet for tablet 864f06b09c484e3a960d149550db97b6 (DEFAULT_TABLE table=default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:58.847398 4248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 864f06b09c484e3a960d149550db97b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:58.848495 3683 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_5aeb666f1fb64981ba79c4f3f4e2b82f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
01:57:58.846 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_393cb35288b14df58c97c418d0db05b6 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
W20260616 01:57:58.853024 3684 catalog_manager.cc:2195] Already present: failed to create HMS catalog entry for table [id=22839c8ea68e457a8280d9f7e1629a0a]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_393cb35288b14df58c97c418d0db05b6 already exists)
I20260616 01:57:58.858868 3684 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_6b418cd896f440f4935c88fdc0a52f51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:58.883437 4381 tablet_bootstrap.cc:492] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Bootstrap starting.
I20260616 01:57:58.889391 3682 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_f8426b357402458d974f69a33f790dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:58.897637 4380 tablet_bootstrap.cc:492] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Bootstrap starting.
I20260616 01:57:58.918969 4382 tablet_bootstrap.cc:492] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Bootstrap starting.
I20260616 01:57:58.901187 4381 tablet_bootstrap.cc:654] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:58.925792 4380 tablet_bootstrap.cc:654] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:58.929612 4380 log.cc:826] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Log is configured to *not* fsync() on all Append() calls
I20260616 01:57:58.926194 4381 log.cc:826] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Log is configured to *not* fsync() on all Append() calls
I20260616 01:57:58.944175 4382 tablet_bootstrap.cc:654] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:58.946643 4382 log.cc:826] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Log is configured to *not* fsync() on all Append() calls
I20260616 01:57:58.954481 4380 tablet_bootstrap.cc:492] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: No bootstrap required, opened a new log
I20260616 01:57:58.955302 4380 ts_tablet_manager.cc:1403] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Time spent bootstrapping tablet: real 0.075s user 0.019s sys 0.003s
I20260616 01:57:58.958580 4381 tablet_bootstrap.cc:492] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: No bootstrap required, opened a new log
I20260616 01:57:58.959101 4381 ts_tablet_manager.cc:1403] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Time spent bootstrapping tablet: real 0.078s user 0.013s sys 0.005s
I20260616 01:57:58.972779 4382 tablet_bootstrap.cc:492] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: No bootstrap required, opened a new log
I20260616 01:57:58.973338 4382 ts_tablet_manager.cc:1403] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.013s
I20260616 01:57:58.989854 3687 catalog_manager.cc:2199] created HMS catalog entry for table [id=c198a7e9a1424db1b682885a74a7f019]
F20260616 01:57:59.017311 3848 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1781575078, eventType=CREATE_TABLE, dbName=default, tableName=table_59ad6b6ba51b44858e1003fece56722a, message={"server":"","servicePrincipal":"","db":"default","table":"table_59ad6b6ba51b44858e1003fece56722a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_59ad6b6ba51b44858e1003fece56722a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1781575078},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/warehouse/table_59ad6b6ba51b44858e1003fece56722a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_59ad6b6ba51b44858e1003fece56722a\",\"kudu.table_id\":\"6035dfb0b8d94f3c981a30d83469e645\",\"transient_lastDdlTime\":\"1781575078\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"640ff0fe2fa14eee9f5b81f68327cf28\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1781575078,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1781575078, eventType=CREATE_TABLE, dbName=default, tableName=table_393cb35288b14df58c97c418d0db05b6, message={"server":"","servicePrincipal":"","db":"default","table":"table_393cb35288b14df58c97c418d0db05b6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_393cb35288b14df58c97c418d0db05b6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1781575077},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/warehouse/table_393cb35288b14df58c97c418d0db05b6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_393cb35288b14df58c97c418d0db05b6\",\"kudu.table_id\":\"c7d302b2976745e38d57e1eaacaf6783\",\"transient_lastDdlTime\":\"1781575077\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"640ff0fe2fa14eee9f5b81f68327cf28\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1781575078,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260616 01:57:59.025144 3687 catalog_manager.cc:2532] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36708:
table { table_name: "default.Table_3692a87561694c028a780ad53630b478" } modify_external_catalogs: true
I20260616 01:57:59.026010 3687 catalog_manager.cc:2780] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36708:
table { table_name: "default.Table_3692a87561694c028a780ad53630b478" } modify_external_catalogs: true
I20260616 01:57:59.050567 4106 tablet_service.cc:1511] Processing CreateTablet for tablet a5cc9e85d93445f3a1f26a12916e04f3 (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:59.051688 4106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5cc9e85d93445f3a1f26a12916e04f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:59.054986 4316 heartbeater.cc:499] Master 127.3.56.61:35255 was elected leader, sending a full tablet report...
I20260616 01:57:59.057456 4381 raft_consensus.cc:359] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.068425 4382 raft_consensus.cc:359] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.058525 3957 tablet_service.cc:1511] Processing CreateTablet for tablet a5cc9e85d93445f3a1f26a12916e04f3 (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:59.069563 4381 raft_consensus.cc:385] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.069974 4381 raft_consensus.cc:740] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.070128 3957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5cc9e85d93445f3a1f26a12916e04f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:59.071085 4381 consensus_queue.cc:260] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.083169 4380 raft_consensus.cc:359] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.084465 4380 raft_consensus.cc:385] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.084911 4380 raft_consensus.cc:740] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.086285 4380 consensus_queue.cc:260] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.098941 4172 heartbeater.cc:499] Master 127.3.56.61:35255 was elected leader, sending a full tablet report...
I20260616 01:57:59.083084 4249 tablet_service.cc:1511] Processing CreateTablet for tablet a5cc9e85d93445f3a1f26a12916e04f3 (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:57:59.103150 4025 heartbeater.cc:499] Master 127.3.56.61:35255 was elected leader, sending a full tablet report...
I20260616 01:57:59.103439 4249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5cc9e85d93445f3a1f26a12916e04f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:57:59.104247 4380 ts_tablet_manager.cc:1434] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Time spent starting tablet: real 0.149s user 0.032s sys 0.012s
I20260616 01:57:59.105315 4380 tablet_bootstrap.cc:492] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: Bootstrap starting.
@ 0x7fe961425950 google::LogMessage::Flush() at ??:0
I20260616 01:57:59.119403 4380 tablet_bootstrap.cc:654] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:59.120422 4382 raft_consensus.cc:385] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.120759 4382 raft_consensus.cc:740] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.121914 4382 consensus_queue.cc:260] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.125008 4381 ts_tablet_manager.cc:1434] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Time spent starting tablet: real 0.166s user 0.023s sys 0.019s
I20260616 01:57:59.126005 4381 tablet_bootstrap.cc:492] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Bootstrap starting.
I20260616 01:57:59.129786 4382 ts_tablet_manager.cc:1434] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Time spent starting tablet: real 0.156s user 0.033s sys 0.001s
I20260616 01:57:59.134554 4382 tablet_bootstrap.cc:492] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: Bootstrap starting.
I20260616 01:57:59.147092 4382 tablet_bootstrap.cc:654] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: Neither blocks nor log segments found. Creating new log.
@ 0x7fe9614256bf google::LogMessage::~LogMessage() at ??:0
I20260616 01:57:59.150502 4381 tablet_bootstrap.cc:654] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:59.154052 4380 tablet_bootstrap.cc:492] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: No bootstrap required, opened a new log
I20260616 01:57:59.154624 4380 ts_tablet_manager.cc:1403] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.008s
I20260616 01:57:59.159184 4382 tablet_bootstrap.cc:492] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: No bootstrap required, opened a new log
I20260616 01:57:59.158737 4380 raft_consensus.cc:359] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.159698 4382 ts_tablet_manager.cc:1403] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.005s
I20260616 01:57:59.159682 4380 raft_consensus.cc:385] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.160027 4380 raft_consensus.cc:740] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.161108 4380 consensus_queue.cc:260] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.163074 4382 raft_consensus.cc:359] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.163715 4382 raft_consensus.cc:385] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.164003 4382 raft_consensus.cc:740] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.164660 4382 consensus_queue.cc:260] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.170950 4380 ts_tablet_manager.cc:1434] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: Time spent starting tablet: real 0.016s user 0.009s sys 0.000s
I20260616 01:57:59.171797 4380 tablet_bootstrap.cc:492] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: Bootstrap starting.
I20260616 01:57:59.172170 4382 ts_tablet_manager.cc:1434] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260616 01:57:59.172997 4382 tablet_bootstrap.cc:492] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: Bootstrap starting.
I20260616 01:57:59.178876 4382 tablet_bootstrap.cc:654] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:59.191973 4381 tablet_bootstrap.cc:492] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: No bootstrap required, opened a new log
I20260616 01:57:59.192401 4381 ts_tablet_manager.cc:1403] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.012s
I20260616 01:57:59.192358 4380 tablet_bootstrap.cc:654] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:59.195170 4381 raft_consensus.cc:359] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.195853 4381 raft_consensus.cc:385] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.196118 4381 raft_consensus.cc:740] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.196772 4382 tablet_bootstrap.cc:492] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: No bootstrap required, opened a new log
I20260616 01:57:59.197185 4382 ts_tablet_manager.cc:1403] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.003s
I20260616 01:57:59.196781 4381 consensus_queue.cc:260] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.199185 4381 ts_tablet_manager.cc:1434] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260616 01:57:59.200006 4381 tablet_bootstrap.cc:492] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: Bootstrap starting.
I20260616 01:57:59.200740 4382 raft_consensus.cc:359] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.201457 4382 raft_consensus.cc:385] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.201736 4382 raft_consensus.cc:740] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.202414 4382 consensus_queue.cc:260] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.208894 4381 tablet_bootstrap.cc:654] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: Neither blocks nor log segments found. Creating new log.
I20260616 01:57:59.213980 4382 ts_tablet_manager.cc:1434] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260616 01:57:59.217466 4380 tablet_bootstrap.cc:492] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: No bootstrap required, opened a new log
I20260616 01:57:59.217940 4380 ts_tablet_manager.cc:1403] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.009s
I20260616 01:57:59.220662 4380 raft_consensus.cc:359] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.221484 4380 raft_consensus.cc:385] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.221804 4380 raft_consensus.cc:740] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Initialized, Role: FOLLOWER
@ 0x7fe96c54b79c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260616 01:57:59.224539 4381 tablet_bootstrap.cc:492] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: No bootstrap required, opened a new log
I20260616 01:57:59.224962 4381 ts_tablet_manager.cc:1403] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.007s
I20260616 01:57:59.226807 3680 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36708:
name: "default.Table_1998ef6052c949edb06e71aa282d6a3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:57:59.227883 4381 raft_consensus.cc:359] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.228590 4381 raft_consensus.cc:385] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:57:59.228879 4381 raft_consensus.cc:740] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Initialized, Role: FOLLOWER
I20260616 01:57:59.222594 4380 consensus_queue.cc:260] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.231174 4380 ts_tablet_manager.cc:1434] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260616 01:57:59.229580 4381 consensus_queue.cc:260] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
@ 0x7fe96c54af5c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260616 01:57:59.257025 4381 ts_tablet_manager.cc:1434] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: Time spent starting tablet: real 0.032s user 0.004s sys 0.003s
W20260616 01:57:59.268816 4322 tablet.cc:2406] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7fe96c54efd2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
W20260616 01:57:59.281704 4178 tablet.cc:2406] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7fe96c54ef8a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
W20260616 01:57:59.315531 4033 tablet.cc:2406] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7fe96c54ef1a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fe96c54eee2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fe96c54e1de std::__1::__function::__func<>::operator()() at ??:0
@ 0x5b4a25 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5b485a std::__1::function<>::operator()() at ??:?
@ 0x7fe96364bfbb kudu::Thread::SuperviseThread() at ??:0
@ 0x47a360 __tsan_thread_start_func at ??:?
@ 0x7fe964d696db start_thread at ??:0
@ 0x7fe9600ca61f clone at ??:0
I20260616 01:57:59.544087 4411 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 321530d69d794e8f94c5424224a44115)
I20260616 01:57:59.544647 4411 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:57:59.546885 4411 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
W20260616 01:57:59.557638 3599 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239): Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
I20260616 01:57:59.590030 4388 raft_consensus.cc:493] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:57:59.590624 4388 raft_consensus.cc:515] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.593916 4388 leader_election.cc:290] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:59.611812 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "864f06b09c484e3a960d149550db97b6" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d" is_pre_election: true
I20260616 01:57:59.612702 3979 raft_consensus.cc:2468] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 0.
I20260616 01:57:59.614225 4058 leader_election.cc:304] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:57:59.615141 4388 raft_consensus.cc:2804] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:57:59.615499 4388 raft_consensus.cc:493] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:57:59.615804 4388 raft_consensus.cc:3060] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.622516 4388 raft_consensus.cc:515] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.624284 4270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "864f06b09c484e3a960d149550db97b6" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110" is_pre_election: true
I20260616 01:57:59.625125 4270 raft_consensus.cc:2468] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 0.
I20260616 01:57:59.625962 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "864f06b09c484e3a960d149550db97b6" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d"
I20260616 01:57:59.626482 3979 raft_consensus.cc:3060] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.631588 4270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "864f06b09c484e3a960d149550db97b6" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110"
I20260616 01:57:59.632086 4270 raft_consensus.cc:3060] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.633482 3979 raft_consensus.cc:2468] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 1.
I20260616 01:57:59.635687 4058 leader_election.cc:304] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:57:59.638686 4270 raft_consensus.cc:2468] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 1.
I20260616 01:57:59.639230 4388 leader_election.cc:290] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 election: Requested vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:59.639814 4388 raft_consensus.cc:2804] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader election won for term 1
I20260616 01:57:59.641260 4388 raft_consensus.cc:697] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 1 LEADER]: Becoming Leader. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Running, Role: LEADER
I20260616 01:57:59.642865 4388 consensus_queue.cc:237] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.655473 4391 raft_consensus.cc:493] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:57:59.655956 4391 raft_consensus.cc:515] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.659247 4391 leader_election.cc:290] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:57:59.676103 4388 raft_consensus.cc:493] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:57:59.676973 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d" is_pre_election: true
I20260616 01:57:59.676777 4388 raft_consensus.cc:515] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.677635 3979 raft_consensus.cc:2468] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3254c8a756fb47a78c049537dbe5e110 in term 0.
I20260616 01:57:59.678910 4202 leader_election.cc:304] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110, dc87c967de1144a6986d9ff87150d21d; no voters:
I20260616 01:57:59.679551 4388 leader_election.cc:290] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:59.680130 4391 raft_consensus.cc:2804] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:57:59.680502 4391 raft_consensus.cc:493] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:57:59.680850 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d" is_pre_election: true
I20260616 01:57:59.680962 4391 raft_consensus.cc:3060] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.681421 3979 raft_consensus.cc:2468] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 0.
I20260616 01:57:59.682618 4058 leader_election.cc:304] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:57:59.683404 4388 raft_consensus.cc:2804] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:57:59.683728 4388 raft_consensus.cc:493] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:57:59.684038 4388 raft_consensus.cc:3060] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.688202 4391 raft_consensus.cc:515] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.690493 4388 raft_consensus.cc:515] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.690920 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d"
I20260616 01:57:59.691418 3979 raft_consensus.cc:3060] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.691681 4391 leader_election.cc:290] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 election: Requested vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:57:59.693527 4388 leader_election.cc:290] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 election: Requested vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:59.693527 3978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d"
I20260616 01:57:59.697502 4270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110" is_pre_election: true
I20260616 01:57:59.697732 4269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110"
I20260616 01:57:59.698354 4270 raft_consensus.cc:2393] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f4d208b546004421a8157a05464cbe7e in current term 1: Already voted for candidate 3254c8a756fb47a78c049537dbe5e110 in this term.
I20260616 01:57:59.699532 3979 raft_consensus.cc:2468] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3254c8a756fb47a78c049537dbe5e110 in term 1.
I20260616 01:57:59.700558 4060 leader_election.cc:304] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4d208b546004421a8157a05464cbe7e; no voters: 3254c8a756fb47a78c049537dbe5e110, dc87c967de1144a6986d9ff87150d21d
I20260616 01:57:59.700762 4202 leader_election.cc:304] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110, dc87c967de1144a6986d9ff87150d21d; no voters:
I20260616 01:57:59.701613 4388 raft_consensus.cc:2749] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260616 01:57:59.702369 4391 raft_consensus.cc:2804] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Leader election won for term 1
W20260616 01:57:59.704046 4328 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.56.61:35255)
W20260616 01:57:59.705055 4328 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.56.61:35255)
W20260616 01:57:59.721793 4061 connection.cc:573] client connection to 127.3.56.61:35255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:57:59.723867 3600 connection.cc:573] client connection to 127.3.56.61:35255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260616 01:57:59.727085 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e" is_pre_election: true
W20260616 01:57:59.727721 4412 negotiation.cc:336] Failed RPC negotiation. Trace:
0616 01:57:59.725762 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.3.56.61:35255 (local address 127.0.0.1:36716)
0616 01:57:59.726695 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0616 01:57:59.726873 (+ 178us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":308}
I20260616 01:57:59.728039 4126 raft_consensus.cc:2393] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3254c8a756fb47a78c049537dbe5e110 in current term 1: Already voted for candidate f4d208b546004421a8157a05464cbe7e in this term.
W20260616 01:57:59.728369 3600 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection reset by peer (error 104)
I20260616 01:57:59.728744 3600 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:57:59.729475 4411 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260616 01:57:59.729676 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a719e057ca84bbb9dcb3a8320777481" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e"
I20260616 01:57:59.730316 4126 raft_consensus.cc:2393] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3254c8a756fb47a78c049537dbe5e110 in current term 1: Already voted for candidate f4d208b546004421a8157a05464cbe7e in this term.
I20260616 01:57:59.719738 4391 raft_consensus.cc:697] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 1 LEADER]: Becoming Leader. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Running, Role: LEADER
I20260616 01:57:59.732194 4391 consensus_queue.cc:237] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.749154 4424 raft_consensus.cc:493] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:57:59.749714 4424 raft_consensus.cc:515] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.758240 4424 leader_election.cc:290] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:59.769632 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5cc9e85d93445f3a1f26a12916e04f3" candidate_uuid: "dc87c967de1144a6986d9ff87150d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e" is_pre_election: true
I20260616 01:57:59.770422 4126 raft_consensus.cc:2468] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc87c967de1144a6986d9ff87150d21d in term 0.
I20260616 01:57:59.776494 3912 leader_election.cc:304] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:57:59.781450 4424 raft_consensus.cc:2804] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:57:59.781890 4424 raft_consensus.cc:493] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:57:59.782430 4424 raft_consensus.cc:3060] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.797098 4424 raft_consensus.cc:515] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:57:59.797499 4270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5cc9e85d93445f3a1f26a12916e04f3" candidate_uuid: "dc87c967de1144a6986d9ff87150d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110" is_pre_election: true
I20260616 01:57:59.798080 4270 raft_consensus.cc:2468] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc87c967de1144a6986d9ff87150d21d in term 0.
I20260616 01:57:59.799850 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5cc9e85d93445f3a1f26a12916e04f3" candidate_uuid: "dc87c967de1144a6986d9ff87150d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e"
I20260616 01:57:59.800395 4126 raft_consensus.cc:3060] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.801139 4270 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5cc9e85d93445f3a1f26a12916e04f3" candidate_uuid: "dc87c967de1144a6986d9ff87150d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110"
I20260616 01:57:59.801654 4270 raft_consensus.cc:3060] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:57:59.714102 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.60:37239
--webserver_interface=127.3.56.60
--webserver_port=34221
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260616 01:57:59.721987 4328 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.56.61:35255)
I20260616 01:57:59.808013 4126 raft_consensus.cc:2468] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc87c967de1144a6986d9ff87150d21d in term 1.
I20260616 01:57:59.809168 3912 leader_election.cc:304] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:57:59.812542 4424 leader_election.cc:290] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 election: Requested vote from peers f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:57:59.812646 4427 raft_consensus.cc:2804] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Leader election won for term 1
I20260616 01:57:59.813223 4427 raft_consensus.cc:697] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 1 LEADER]: Becoming Leader. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Running, Role: LEADER
I20260616 01:57:59.813301 4270 raft_consensus.cc:2468] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc87c967de1144a6986d9ff87150d21d in term 1.
W20260616 01:57:59.806190 4328 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.AlterTable -> {remote=127.3.56.61:35255, user_credentials={real_user=slave}} blocked reactor thread for 84723us
I20260616 01:57:59.816691 4427 consensus_queue.cc:237] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
W20260616 01:57:59.824877 4328 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.56.61:35255)
W20260616 01:58:00.148674 4428 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:58:00.149313 4428 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:58:00.149799 4428 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260616 01:58:00.184521 4428 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260616 01:58:00.184859 4428 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:58:00.185111 4428 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260616 01:58:00.185429 4428 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260616 01:58:00.227104 4428 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.60:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.56.60
--webserver_port=34221
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:58:00.228503 4428 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:58:00.230186 4428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260616 01:58:00.241075 4439 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 321530d69d794e8f94c5424224a44115)
I20260616 01:58:00.241876 4439 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
W20260616 01:58:00.243587 4438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:58:00.250376 4439 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
W20260616 01:58:00.273451 3599 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239): Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
W20260616 01:58:00.276263 3600 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:00.276997 3600 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:58:00.277951 4439 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260616 01:58:00.647832 4451 consensus_queue.cc:1048] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260616 01:58:00.725184 4453 consensus_queue.cc:1048] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260616 01:58:00.889443 4460 consensus_queue.cc:1048] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260616 01:58:00.929337 4462 consensus_queue.cc:1048] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260616 01:58:00.930405 4058 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.56.1:34353, user_credentials={real_user=slave}} blocked reactor thread for 63960.2us
I20260616 01:58:00.994069 4452 consensus_queue.cc:1048] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260616 01:58:00.999342 4465 consensus_queue.cc:1048] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260616 01:58:01.127429 4327 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.3.56.60:37239, user_credentials={real_user=slave}} blocked reactor thread for 79626.7us
W20260616 01:58:01.129630 4326 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.3.56.62:35519, user_credentials={real_user=slave}} blocked reactor thread for 78295.2us
I20260616 01:58:01.188305 4470 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 321530d69d794e8f94c5424224a44115)
I20260616 01:58:01.260660 4470 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:01.395169 4470 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
W20260616 01:58:01.387274 3913 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.56.3:41531, user_credentials={real_user=slave}} blocked reactor thread for 66215.3us
W20260616 01:58:01.441062 3599 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239): Network error: Client connection negotiation failed: client connection to 127.3.56.60:37239: connect: Connection refused (error 111)
W20260616 01:58:01.458380 3600 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:01.459203 3600 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115, 673bf665c7d743579440c02560de26b9
I20260616 01:58:01.463070 4470 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260616 01:58:01.644174 4437 debug-util.cc:398] Leaking SignalData structure 0x7b08000376c0 after lost signal to thread 4428
W20260616 01:58:01.845180 4437 kernel_stack_watchdog.cc:198] Thread 4428 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260616 01:58:00.243629 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:58:01.846611 4428 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.604s user 0.503s sys 1.023s
W20260616 01:58:01.846992 4428 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.605s user 0.503s sys 1.023s
W20260616 01:58:01.847560 4442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:58:01.850546 4441 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1607 milliseconds
I20260616 01:58:01.850574 4428 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260616 01:58:01.851912 4428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:58:01.854673 4428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:58:01.856038 4428 hybrid_clock.cc:648] HybridClock initialized: now 1781575081856003 us; error 31 us; skew 500 ppm
I20260616 01:58:01.856830 4428 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:58:01.862951 4428 webserver.cc:533] Webserver started at http://127.3.56.60:34221/ using document root <none> and password file <none>
I20260616 01:58:01.863839 4428 fs_manager.cc:362] Metadata directory not provided
I20260616 01:58:01.864076 4428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:58:01.871604 4428 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260616 01:58:01.876336 4485 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:58:01.877578 4428 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260616 01:58:01.878098 4428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
uuid: "673bf665c7d743579440c02560de26b9"
format_stamp: "Formatted at 2026-06-16 01:57:48 on dist-test-slave-s7k7"
I20260616 01:58:01.880163 4428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:58:01.961760 4428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:58:01.963871 4428 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:58:01.964452 4428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:58:02.071844 4428 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.60:37239
I20260616 01:58:02.071924 4539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.60:37239 every 8 connection(s)
I20260616 01:58:02.074687 4428 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
I20260616 01:58:02.079099 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 4428
I20260616 01:58:02.092902 4540 sys_catalog.cc:263] Verifying existing consensus state
I20260616 01:58:02.100401 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Bootstrap starting.
I20260616 01:58:02.114607 4540 log.cc:826] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Log is configured to *not* fsync() on all Append() calls
I20260616 01:58:02.128785 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260616 01:58:02.129527 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Bootstrap complete.
I20260616 01:58:02.147751 4540 raft_consensus.cc:359] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:02.149906 4540 raft_consensus.cc:740] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673bf665c7d743579440c02560de26b9, State: Initialized, Role: FOLLOWER
I20260616 01:58:02.150588 4540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:02.152702 4545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:58:02.153347 4545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:58:02.154062 4540 sys_catalog.cc:565] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260616 01:58:02.200184 4560 catalog_manager.cc:1294] Loaded cluster ID: 640ff0fe2fa14eee9f5b81f68327cf28
I20260616 01:58:02.200475 4560 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: loading cluster ID for follower catalog manager: success
I20260616 01:58:02.208949 4560 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: acquiring CA information for follower catalog manager: success
I20260616 01:58:02.215448 4560 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260616 01:58:02.211077 4428 master_runner.cc:187] Error getting master registration for 127.3.56.61:35255: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111),
I20260616 01:58:02.231105 4428 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:02.264827 4171 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:58:02.271938 4501 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f4d208b546004421a8157a05464cbe7e" instance_seqno: 1781575075463092) as {username='slave'} at 127.3.56.2:37451; Asking this server to re-register.
I20260616 01:58:02.273746 4171 heartbeater.cc:461] Registering TS with master...
I20260616 01:58:02.277515 4501 ts_manager.cc:194] Registered new tserver with Master: f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:58:02.406450 4315 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:58:02.406901 4315 heartbeater.cc:499] Master 127.3.56.60:37239 was elected leader, sending a full tablet report...
I20260616 01:58:02.410598 4501 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" instance_seqno: 1781575077465778) as {username='slave'} at 127.3.56.3:46583; Asking this server to re-register.
I20260616 01:58:02.411900 4315 heartbeater.cc:461] Registering TS with master...
I20260616 01:58:02.416909 4501 ts_manager.cc:194] Registered new tserver with Master: 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:58:02.466029 4024 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:58:02.469486 4501 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" instance_seqno: 1781575072950480) as {username='slave'} at 127.3.56.1:43557; Asking this server to re-register.
I20260616 01:58:02.470654 4024 heartbeater.cc:461] Registering TS with master...
I20260616 01:58:02.473964 4501 ts_manager.cc:194] Registered new tserver with Master: dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353)
I20260616 01:58:02.533233 4567 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 321530d69d794e8f94c5424224a44115)
I20260616 01:58:02.533726 4567 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:02.535653 4567 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
W20260616 01:58:02.567169 3600 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:02.568169 4512 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9" is_pre_election: true
I20260616 01:58:02.568982 4512 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f509c7b2b84e4bd1a6e09d901dd331b7 in term 1.
I20260616 01:58:02.570312 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115
I20260616 01:58:02.571048 4567 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260616 01:58:02.571383 4567 raft_consensus.cc:493] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 321530d69d794e8f94c5424224a44115)
I20260616 01:58:02.571705 4567 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 2 FOLLOWER]: Advancing to term 3
I20260616 01:58:02.576918 4567 raft_consensus.cc:515] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:02.578564 4567 leader_election.cc:290] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 election: Requested vote from peers 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255), 673bf665c7d743579440c02560de26b9 (127.3.56.60:37239)
I20260616 01:58:02.579689 4512 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "673bf665c7d743579440c02560de26b9"
I20260616 01:58:02.580199 4512 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 1 FOLLOWER]: Advancing to term 3
W20260616 01:58:02.582188 3600 leader_election.cc:336] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:02.587124 4512 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f509c7b2b84e4bd1a6e09d901dd331b7 in term 3.
I20260616 01:58:02.588001 3599 leader_election.cc:304] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 673bf665c7d743579440c02560de26b9, f509c7b2b84e4bd1a6e09d901dd331b7; no voters: 321530d69d794e8f94c5424224a44115
I20260616 01:58:02.588618 4567 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 3 FOLLOWER]: Leader election won for term 3
I20260616 01:58:02.589890 4567 raft_consensus.cc:697] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [term 3 LEADER]: Becoming Leader. State: Replica: f509c7b2b84e4bd1a6e09d901dd331b7, State: Running, Role: LEADER
I20260616 01:58:02.590867 4567 consensus_queue.cc:237] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:02.597658 4571 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f509c7b2b84e4bd1a6e09d901dd331b7. Latest consensus state: current_term: 3 leader_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:58:02.598469 4571 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: This master's current role is: LEADER
I20260616 01:58:02.599998 4574 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
I20260616 01:58:02.605616 4574 catalog_manager.cc:679] Loaded metadata for table default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]
I20260616 01:58:02.607165 4574 catalog_manager.cc:679] Loaded metadata for table default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]
I20260616 01:58:02.608317 4574 catalog_manager.cc:679] Loaded metadata for table default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]
I20260616 01:58:02.612641 4574 tablet_loader.cc:96] loaded metadata for tablet 3a719e057ca84bbb9dcb3a8320777481 (table default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645])
I20260616 01:58:02.613591 4574 tablet_loader.cc:96] loaded metadata for tablet 864f06b09c484e3a960d149550db97b6 (table default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17])
I20260616 01:58:02.614436 4574 tablet_loader.cc:96] loaded metadata for tablet a5cc9e85d93445f3a1f26a12916e04f3 (table default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019])
I20260616 01:58:02.615324 4574 catalog_manager.cc:1519] Initializing Kudu cluster ID...
I20260616 01:58:02.618269 4574 catalog_manager.cc:1294] Loaded cluster ID: 640ff0fe2fa14eee9f5b81f68327cf28
I20260616 01:58:02.618487 4574 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
I20260616 01:58:02.622965 4574 catalog_manager.cc:1539] Loading token signing keys...
I20260616 01:58:02.626052 4574 catalog_manager.cc:6080] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Loaded TSK: 0
I20260616 01:58:02.626976 4574 catalog_manager.cc:1549] Initializing in-progress tserver states...
I20260616 01:58:02.629587 4574 catalog_manager.cc:1562] Loading latest processed Hive Metastore notification log event ID...
I20260616 01:58:02.632050 4574 catalog_manager.cc:5918] Last processed Hive Metastore notification event ID: -1
I20260616 01:58:02.698449 3606 master_service.cc:960] ReplaceTablet: received request to replace tablet a5cc9e85d93445f3a1f26a12916e04f3 from {username='slave'} at 127.0.0.1:54444
I20260616 01:58:02.698843 3605 master_service.cc:960] ReplaceTablet: received request to replace tablet 864f06b09c484e3a960d149550db97b6 from {username='slave'} at 127.0.0.1:54444
I20260616 01:58:02.693987 3613 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54444:
name: "default.Table_1998ef6052c949edb06e71aa282d6a3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:58:02.695020 3611 catalog_manager.cc:2532] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54444:
table { table_name: "default.Table_3692a87561694c028a780ad53630b478" } modify_external_catalogs: true
I20260616 01:58:02.696249 3609 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54444:
name: "default.Table_5aeb666f1fb64981ba79c4f3f4e2b82f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:58:02.722388 3611 catalog_manager.cc:2780] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54444:
table { table_name: "default.Table_3692a87561694c028a780ad53630b478" } modify_external_catalogs: true
I20260616 01:58:02.694571 3612 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54444:
name: "default.Table_f8426b357402458d974f69a33f790dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260616 01:58:02.697397 3608 catalog_manager.cc:3569] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54444:
table { table_name: "default.Table_59ad6b6ba51b44858e1003fece56722a" } new_table_name: "default.Table_100c8bee45e34b27a94e7835f0878e64" modify_external_catalogs: true
I20260616 01:58:02.697768 3607 catalog_manager.cc:3569] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54444:
table { table_name: "default.Table_57c462b8048f489fac3e5529096d3f59" } new_table_name: "default.Table_5afd5bde5bcd4fe3928ccb78ecfa3824" modify_external_catalogs: true
I20260616 01:58:02.695909 3610 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54444:
name: "default.Table_6b418cd896f440f4935c88fdc0a52f51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260616 01:58:02.730248 3883 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1781575078, eventType=CREATE_TABLE, dbName=default, tableName=table_59ad6b6ba51b44858e1003fece56722a, message={"server":"","servicePrincipal":"","db":"default","table":"table_59ad6b6ba51b44858e1003fece56722a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_59ad6b6ba51b44858e1003fece56722a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1781575078},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/warehouse/table_59ad6b6ba51b44858e1003fece56722a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_59ad6b6ba51b44858e1003fece56722a\",\"kudu.table_id\":\"6035dfb0b8d94f3c981a30d83469e645\",\"transient_lastDdlTime\":\"1781575078\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"640ff0fe2fa14eee9f5b81f68327cf28\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1781575078,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1781575078, eventType=CREATE_TABLE, dbName=default, tableName=table_393cb35288b14df58c97c418d0db05b6, message={"server":"","servicePrincipal":"","db":"default","table":"table_393cb35288b14df58c97c418d0db05b6","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_393cb35288b14df58c97c418d0db05b6\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1781575077},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/warehouse/table_393cb35288b14df58c97c418d0db05b6\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_393cb35288b14df58c97c418d0db05b6\",\"kudu.table_id\":\"c7d302b2976745e38d57e1eaacaf6783\",\"transient_lastDdlTime\":\"1781575077\",\"comment\":\"\",\"kudu.master_addresses\":\"127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"640ff0fe2fa14eee9f5b81f68327cf28\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1781575078,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
W20260616 01:58:02.737033 3600 consensus_peers.cc:597] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 -> Peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Couldn't send request to peer 321530d69d794e8f94c5424224a44115. Status: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260616 01:58:02.739246 4512 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Refusing update from remote peer f509c7b2b84e4bd1a6e09d901dd331b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260616 01:58:02.740638 4567 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7f05ccc28950 google::LogMessage::Flush() at ??:0
@ 0x7f05ccc286bf google::LogMessage::~LogMessage() at ??:0
I20260616 01:58:02.785884 4576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f509c7b2b84e4bd1a6e09d901dd331b7. Latest consensus state: current_term: 3 leader_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:58:02.786612 4576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:58:02.786849 4577 mvcc.cc:204] Tried to move back new op lower bound from 7297331523698421760 to 7297331519817850880. Current Snapshot: MvccSnapshot[applied={T|T < 7297331523698421760 or (T in {7297331523698421760})}]
I20260616 01:58:02.787726 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:58:02.788473 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:58:02.792768 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:58:02.793496 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:58:02.794484 3605 catalog_manager.cc:5983] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Sending DeleteTablet for 3 replicas of tablet 864f06b09c484e3a960d149550db97b6
I20260616 01:58:02.806280 3606 catalog_manager.cc:5983] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Sending DeleteTablet for 3 replicas of tablet a5cc9e85d93445f3a1f26a12916e04f3
I20260616 01:58:02.807428 4571 sys_catalog.cc:455] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:58:02.808074 4571 sys_catalog.cc:458] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7 [sys.catalog]: This master's current role is: LEADER
I20260616 01:58:02.808571 3606 catalog_manager.cc:6784] ReplaceTablet: tablet a5cc9e85d93445f3a1f26a12916e04f3 deleted and replaced by tablet cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:02.810760 3605 catalog_manager.cc:6784] ReplaceTablet: tablet 864f06b09c484e3a960d149550db97b6 deleted and replaced by tablet 1a2585bbddfd44aa95fdde7b0b52e0fd
@ 0x7f05d7d4e79c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f05d7d4df5c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260616 01:58:02.888262 3957 tablet_service.cc:1558] Processing DeleteTablet for tablet 864f06b09c484e3a960d149550db97b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a2585bbddfd44aa95fdde7b0b52e0fd) from {username='slave'} at 127.0.0.1:34702
I20260616 01:58:02.890489 3958 tablet_service.cc:1558] Processing DeleteTablet for tablet a5cc9e85d93445f3a1f26a12916e04f3 with delete_type TABLET_DATA_DELETED (replaced by tablet cf37e86d57d346dfae2e866bed3ef847) from {username='slave'} at 127.0.0.1:34702
@ 0x7f05d7d51fd2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260616 01:58:02.893959 4106 tablet_service.cc:1558] Processing DeleteTablet for tablet 864f06b09c484e3a960d149550db97b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a2585bbddfd44aa95fdde7b0b52e0fd) from {username='slave'} at 127.0.0.1:35402
I20260616 01:58:02.896682 4105 tablet_service.cc:1558] Processing DeleteTablet for tablet a5cc9e85d93445f3a1f26a12916e04f3 with delete_type TABLET_DATA_DELETED (replaced by tablet cf37e86d57d346dfae2e866bed3ef847) from {username='slave'} at 127.0.0.1:35402
I20260616 01:58:02.903355 4593 tablet_replica.cc:333] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: stopping tablet replica
I20260616 01:58:02.904285 4593 raft_consensus.cc:2243] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:02.904886 4593 raft_consensus.cc:2272] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:02.912889 4593 ts_tablet_manager.cc:1916] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:02.913318 4249 tablet_service.cc:1558] Processing DeleteTablet for tablet a5cc9e85d93445f3a1f26a12916e04f3 with delete_type TABLET_DATA_DELETED (replaced by tablet cf37e86d57d346dfae2e866bed3ef847) from {username='slave'} at 127.0.0.1:55854
I20260616 01:58:02.906661 4595 tablet_replica.cc:333] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: stopping tablet replica
I20260616 01:58:02.914572 4595 raft_consensus.cc:2243] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 1 LEADER]: Raft consensus shutting down.
I20260616 01:58:02.915547 4595 raft_consensus.cc:2272] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:02.916126 4248 tablet_service.cc:1558] Processing DeleteTablet for tablet 864f06b09c484e3a960d149550db97b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a2585bbddfd44aa95fdde7b0b52e0fd) from {username='slave'} at 127.0.0.1:55854
I20260616 01:58:02.918892 4595 ts_tablet_manager.cc:1916] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:02.931444 4593 ts_tablet_manager.cc:1929] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:02.931878 4593 log.cc:1199] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal/wals/864f06b09c484e3a960d149550db97b6
I20260616 01:58:02.932955 4593 ts_tablet_manager.cc:1950] T 864f06b09c484e3a960d149550db97b6 P dc87c967de1144a6986d9ff87150d21d: Deleting consensus metadata
I20260616 01:58:02.936412 4593 tablet_replica.cc:333] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: stopping tablet replica
I20260616 01:58:02.937264 4593 raft_consensus.cc:2243] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 1 LEADER]: Raft consensus shutting down.
I20260616 01:58:02.938172 4593 raft_consensus.cc:2272] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:02.940119 4593 ts_tablet_manager.cc:1916] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:02.940281 4595 ts_tablet_manager.cc:1929] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:02.940743 4595 log.cc:1199] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal/wals/864f06b09c484e3a960d149550db97b6
I20260616 01:58:02.941645 4595 ts_tablet_manager.cc:1950] T 864f06b09c484e3a960d149550db97b6 P f4d208b546004421a8157a05464cbe7e: Deleting consensus metadata
I20260616 01:58:02.944684 4595 tablet_replica.cc:333] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: stopping tablet replica
I20260616 01:58:02.945426 4595 raft_consensus.cc:2243] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:02.945948 4595 raft_consensus.cc:2272] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:02.946686 4596 tablet_replica.cc:333] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: stopping tablet replica
I20260616 01:58:02.947831 4595 ts_tablet_manager.cc:1916] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:02.948730 4596 raft_consensus.cc:2243] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:02.948657 3597 catalog_manager.cc:5027] TS dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): tablet 864f06b09c484e3a960d149550db97b6 (table default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]) successfully deleted
W20260616 01:58:02.949713 3644 debug-util.cc:398] Leaking SignalData structure 0x7b08000944e0 after lost signal to thread 3581
I20260616 01:58:02.950757 3598 catalog_manager.cc:5027] TS f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): tablet 864f06b09c484e3a960d149550db97b6 (table default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]) successfully deleted
W20260616 01:58:02.951495 3644 debug-util.cc:398] Leaking SignalData structure 0x7b0800058d60 after lost signal to thread 4478
I20260616 01:58:02.959007 4593 ts_tablet_manager.cc:1929] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:02.959470 4593 log.cc:1199] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal/wals/a5cc9e85d93445f3a1f26a12916e04f3
I20260616 01:58:02.960414 4593 ts_tablet_manager.cc:1950] T a5cc9e85d93445f3a1f26a12916e04f3 P dc87c967de1144a6986d9ff87150d21d: Deleting consensus metadata
I20260616 01:58:02.963366 3597 catalog_manager.cc:5027] TS dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): tablet a5cc9e85d93445f3a1f26a12916e04f3 (table default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]) successfully deleted
I20260616 01:58:02.965323 4596 raft_consensus.cc:2272] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:02.973062 4595 ts_tablet_manager.cc:1929] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:02.973688 4595 log.cc:1199] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal/wals/a5cc9e85d93445f3a1f26a12916e04f3
I20260616 01:58:02.974677 4595 ts_tablet_manager.cc:1950] T a5cc9e85d93445f3a1f26a12916e04f3 P f4d208b546004421a8157a05464cbe7e: Deleting consensus metadata
I20260616 01:58:02.978196 3598 catalog_manager.cc:5027] TS f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): tablet a5cc9e85d93445f3a1f26a12916e04f3 (table default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]) successfully deleted
I20260616 01:58:02.979274 4596 ts_tablet_manager.cc:1916] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:02.978648 3605 catalog_manager.cc:5696] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3). New cstate: current_term: 1 leader_uuid: "3254c8a756fb47a78c049537dbe5e110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } health_report { overall_health: HEALTHY } } }
I20260616 01:58:02.983556 3957 tablet_service.cc:1511] Processing CreateTablet for tablet cf37e86d57d346dfae2e866bed3ef847 (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:02.975998 4249 tablet_service.cc:1511] Processing CreateTablet for tablet cf37e86d57d346dfae2e866bed3ef847 (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:02.984689 4106 tablet_service.cc:1511] Processing CreateTablet for tablet cf37e86d57d346dfae2e866bed3ef847 (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:02.984995 3957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf37e86d57d346dfae2e866bed3ef847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:02.985677 4249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf37e86d57d346dfae2e866bed3ef847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:02.985963 4106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf37e86d57d346dfae2e866bed3ef847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:02.997288 3958 tablet_service.cc:1511] Processing CreateTablet for tablet 1a2585bbddfd44aa95fdde7b0b52e0fd (DEFAULT_TABLE table=default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:02.998672 3958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2585bbddfd44aa95fdde7b0b52e0fd. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f05d7d51f8a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260616 01:58:03.001147 4105 tablet_service.cc:1511] Processing CreateTablet for tablet 1a2585bbddfd44aa95fdde7b0b52e0fd (DEFAULT_TABLE table=default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:03.002348 4105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2585bbddfd44aa95fdde7b0b52e0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:03.002556 4248 tablet_service.cc:1511] Processing CreateTablet for tablet 1a2585bbddfd44aa95fdde7b0b52e0fd (DEFAULT_TABLE table=default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:03.003710 4248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a2585bbddfd44aa95fdde7b0b52e0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:03.014830 4317 heartbeater.cc:499] Master 127.3.56.62:35519 was elected leader, sending a full tablet report...
I20260616 01:58:03.029773 4601 tablet_bootstrap.cc:492] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e: Bootstrap starting.
I20260616 01:58:03.036975 4601 tablet_bootstrap.cc:654] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.047753 4601 tablet_bootstrap.cc:492] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e: No bootstrap required, opened a new log
I20260616 01:58:03.048281 4601 ts_tablet_manager.cc:1403] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.012s
I20260616 01:58:03.051057 4601 raft_consensus.cc:359] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.051919 4601 raft_consensus.cc:385] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.052225 4601 raft_consensus.cc:740] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.053046 4601 consensus_queue.cc:260] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.055287 4173 heartbeater.cc:499] Master 127.3.56.62:35519 was elected leader, sending a full tablet report...
I20260616 01:58:03.061086 4601 ts_tablet_manager.cc:1434] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260616 01:58:03.065016 4603 tablet_bootstrap.cc:492] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: Bootstrap starting.
I20260616 01:58:03.065950 4601 tablet_bootstrap.cc:492] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: Bootstrap starting.
I20260616 01:58:03.071875 4601 tablet_bootstrap.cc:654] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.071964 4604 tablet_bootstrap.cc:492] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110: Bootstrap starting.
I20260616 01:58:03.082949 4601 tablet_bootstrap.cc:492] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: No bootstrap required, opened a new log
I20260616 01:58:03.083468 4601 ts_tablet_manager.cc:1403] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20260616 01:58:03.083626 4566 raft_consensus.cc:493] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:03.084167 4566 raft_consensus.cc:515] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.086325 4566 leader_election.cc:290] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:58:03.086189 4601 raft_consensus.cc:359] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:03.086855 4601 raft_consensus.cc:385] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.087114 4601 raft_consensus.cc:740] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.087987 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a2585bbddfd44aa95fdde7b0b52e0fd" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d" is_pre_election: true
I20260616 01:58:03.087924 4601 consensus_queue.cc:260] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:03.088253 4269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a2585bbddfd44aa95fdde7b0b52e0fd" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110" is_pre_election: true
W20260616 01:58:03.089727 4058 leader_election.cc:343] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Illegal state: must be running to vote when last-logged opid is not known
W20260616 01:58:03.090237 4060 leader_election.cc:343] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531): Illegal state: must be running to vote when last-logged opid is not known
I20260616 01:58:03.090648 4060 leader_election.cc:304] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f4d208b546004421a8157a05464cbe7e; no voters: 3254c8a756fb47a78c049537dbe5e110, dc87c967de1144a6986d9ff87150d21d
I20260616 01:58:03.091405 4604 tablet_bootstrap.cc:654] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.093531 4566 raft_consensus.cc:2749] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:58:03.096609 4601 ts_tablet_manager.cc:1434] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260616 01:58:03.099931 4603 tablet_bootstrap.cc:654] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: Neither blocks nor log segments found. Creating new log.
@ 0x7f05d7d51f1a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260616 01:58:03.115118 3606 master_service.cc:960] ReplaceTablet: received request to replace tablet cf37e86d57d346dfae2e866bed3ef847 from {username='slave'} at 127.0.0.1:54444
I20260616 01:58:03.117888 3605 master_service.cc:960] ReplaceTablet: received request to replace tablet 3a719e057ca84bbb9dcb3a8320777481 from {username='slave'} at 127.0.0.1:54444
I20260616 01:58:03.121902 4604 tablet_bootstrap.cc:492] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110: No bootstrap required, opened a new log
I20260616 01:58:03.122360 4604 ts_tablet_manager.cc:1403] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.004s
I20260616 01:58:03.125144 4604 raft_consensus.cc:359] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.125860 4604 raft_consensus.cc:385] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.126127 4604 raft_consensus.cc:740] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.126809 4604 consensus_queue.cc:260] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
W20260616 01:58:03.127837 3600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260616 01:58:03.128870 4604 ts_tablet_manager.cc:1434] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260616 01:58:03.132145 4603 tablet_bootstrap.cc:492] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: No bootstrap required, opened a new log
I20260616 01:58:03.132591 4603 ts_tablet_manager.cc:1403] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.006s
I20260616 01:58:03.135388 4603 raft_consensus.cc:359] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:03.136042 4603 raft_consensus.cc:385] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.136327 4603 raft_consensus.cc:740] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.137234 4603 consensus_queue.cc:260] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:03.138494 4596 ts_tablet_manager.cc:1929] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:03.138899 4596 log.cc:1199] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal/wals/a5cc9e85d93445f3a1f26a12916e04f3
I20260616 01:58:03.139339 4603 ts_tablet_manager.cc:1434] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260616 01:58:03.139861 4596 ts_tablet_manager.cc:1950] T a5cc9e85d93445f3a1f26a12916e04f3 P 3254c8a756fb47a78c049537dbe5e110: Deleting consensus metadata
I20260616 01:58:03.140177 4603 tablet_bootstrap.cc:492] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d: Bootstrap starting.
I20260616 01:58:03.142439 4030 heartbeater.cc:499] Master 127.3.56.62:35519 was elected leader, sending a full tablet report...
@ 0x7f05d7d51ee2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260616 01:58:03.146732 4604 tablet_bootstrap.cc:492] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110: Bootstrap starting.
I20260616 01:58:03.150391 4596 tablet_replica.cc:333] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: stopping tablet replica
I20260616 01:58:03.151152 4596 raft_consensus.cc:2243] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:03.151656 4596 raft_consensus.cc:2272] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:03.152558 4604 tablet_bootstrap.cc:654] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.153638 4596 ts_tablet_manager.cc:1916] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:03.155349 4603 tablet_bootstrap.cc:654] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.155762 3599 catalog_manager.cc:5027] TS 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531): tablet a5cc9e85d93445f3a1f26a12916e04f3 (table default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]) successfully deleted
@ 0x7f05d7d511de std::__1::__function::__func<>::operator()() at ??:0
@ 0x5b4a25 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5b485a std::__1::function<>::operator()() at ??:?
I20260616 01:58:03.190347 3606 catalog_manager.cc:5983] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Sending DeleteTablet for 3 replicas of tablet cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:03.192395 4248 tablet_service.cc:1558] Processing DeleteTablet for tablet cf37e86d57d346dfae2e866bed3ef847 with delete_type TABLET_DATA_DELETED (replaced by tablet e622147476e046eda3b9a698d57abc5c) from {username='slave'} at 127.0.0.1:55854
@ 0x7f05cee4efbb kudu::Thread::SuperviseThread() at ??:0
@ 0x47a360 __tsan_thread_start_func at ??:?
I20260616 01:58:03.206050 3605 catalog_manager.cc:5983] T 00000000000000000000000000000000 P f509c7b2b84e4bd1a6e09d901dd331b7: Sending DeleteTablet for 3 replicas of tablet 3a719e057ca84bbb9dcb3a8320777481
I20260616 01:58:03.207667 3605 catalog_manager.cc:6784] ReplaceTablet: tablet 3a719e057ca84bbb9dcb3a8320777481 deleted and replaced by tablet de807bc8ba2c4132909082130958776e
I20260616 01:58:03.209246 3957 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a719e057ca84bbb9dcb3a8320777481 with delete_type TABLET_DATA_DELETED (replaced by tablet de807bc8ba2c4132909082130958776e) from {username='slave'} at 127.0.0.1:34702
I20260616 01:58:03.209962 4106 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a719e057ca84bbb9dcb3a8320777481 with delete_type TABLET_DATA_DELETED (replaced by tablet de807bc8ba2c4132909082130958776e) from {username='slave'} at 127.0.0.1:35402
I20260616 01:58:03.210638 4595 tablet_replica.cc:333] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: stopping tablet replica
I20260616 01:58:03.210685 4593 tablet_replica.cc:333] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: stopping tablet replica
I20260616 01:58:03.211331 4595 raft_consensus.cc:2243] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:03.211381 4593 raft_consensus.cc:2243] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:03.211809 4595 raft_consensus.cc:2272] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:03.211869 4593 raft_consensus.cc:2272] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:03.213522 4248 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a719e057ca84bbb9dcb3a8320777481 with delete_type TABLET_DATA_DELETED (replaced by tablet de807bc8ba2c4132909082130958776e) from {username='slave'} at 127.0.0.1:55854
I20260616 01:58:03.213804 4595 ts_tablet_manager.cc:1916] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:03.213937 4593 ts_tablet_manager.cc:1916] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:03.225112 4596 ts_tablet_manager.cc:1929] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:03.225524 4596 log.cc:1199] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal/wals/864f06b09c484e3a960d149550db97b6
I20260616 01:58:03.226241 4596 ts_tablet_manager.cc:1950] T 864f06b09c484e3a960d149550db97b6 P 3254c8a756fb47a78c049537dbe5e110: Deleting consensus metadata
I20260616 01:58:03.227777 3957 tablet_service.cc:1558] Processing DeleteTablet for tablet cf37e86d57d346dfae2e866bed3ef847 with delete_type TABLET_DATA_DELETED (replaced by tablet e622147476e046eda3b9a698d57abc5c) from {username='slave'} at 127.0.0.1:34702
I20260616 01:58:03.228669 3599 catalog_manager.cc:5027] TS 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531): tablet 864f06b09c484e3a960d149550db97b6 (table default.table_57c462b8048f489fac3e5529096d3f59 [id=2ba9c42af02e485bb8990524ac09da17]) successfully deleted
@ 0x7f05d056c6db start_thread at ??:0
I20260616 01:58:03.234107 3606 catalog_manager.cc:6784] ReplaceTablet: tablet cf37e86d57d346dfae2e866bed3ef847 deleted and replaced by tablet e622147476e046eda3b9a698d57abc5c
I20260616 01:58:03.234114 4106 tablet_service.cc:1558] Processing DeleteTablet for tablet cf37e86d57d346dfae2e866bed3ef847 with delete_type TABLET_DATA_DELETED (replaced by tablet e622147476e046eda3b9a698d57abc5c) from {username='slave'} at 127.0.0.1:35402
W20260616 01:58:03.240499 3599 catalog_manager.cc:5007] TS 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531): delete failed for tablet cf37e86d57d346dfae2e866bed3ef847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf37e86d57d346dfae2e866bed3ef847 already in progress: creating tablet
I20260616 01:58:03.241588 4596 tablet_replica.cc:333] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: stopping tablet replica
I20260616 01:58:03.242386 4596 raft_consensus.cc:2243] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 1 LEADER]: Raft consensus shutting down.
I20260616 01:58:03.242792 4593 ts_tablet_manager.cc:1929] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:03.243185 4593 log.cc:1199] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal/wals/3a719e057ca84bbb9dcb3a8320777481
I20260616 01:58:03.243355 4596 raft_consensus.cc:2272] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:03.244005 4593 ts_tablet_manager.cc:1950] T 3a719e057ca84bbb9dcb3a8320777481 P dc87c967de1144a6986d9ff87150d21d: Deleting consensus metadata
I20260616 01:58:03.245314 4596 ts_tablet_manager.cc:1916] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:03.229143 4595 ts_tablet_manager.cc:1929] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:03.258752 4595 log.cc:1199] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal/wals/3a719e057ca84bbb9dcb3a8320777481
I20260616 01:58:03.258792 3597 catalog_manager.cc:5027] TS dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): tablet 3a719e057ca84bbb9dcb3a8320777481 (table default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]) successfully deleted
I20260616 01:58:03.260212 4593 tablet_replica.cc:333] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: stopping tablet replica
I20260616 01:58:03.262595 4593 raft_consensus.cc:2243] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:03.263075 4593 raft_consensus.cc:2272] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:03.259505 4595 ts_tablet_manager.cc:1950] T 3a719e057ca84bbb9dcb3a8320777481 P f4d208b546004421a8157a05464cbe7e: Deleting consensus metadata
I20260616 01:58:03.268244 4595 tablet_replica.cc:333] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: stopping tablet replica
I20260616 01:58:03.268486 4593 ts_tablet_manager.cc:1916] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:03.268930 4595 raft_consensus.cc:2243] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Raft consensus shutting down.
I20260616 01:58:03.268890 3598 catalog_manager.cc:5027] TS f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): tablet 3a719e057ca84bbb9dcb3a8320777481 (table default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]) successfully deleted
I20260616 01:58:03.269398 4595 raft_consensus.cc:2272] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Raft consensus is shut down!
I20260616 01:58:03.275607 3957 tablet_service.cc:1511] Processing CreateTablet for tablet de807bc8ba2c4132909082130958776e (DEFAULT_TABLE table=default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:03.276773 3957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de807bc8ba2c4132909082130958776e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:03.276696 4106 tablet_service.cc:1511] Processing CreateTablet for tablet de807bc8ba2c4132909082130958776e (DEFAULT_TABLE table=default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:03.277873 4106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de807bc8ba2c4132909082130958776e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:03.282612 4595 ts_tablet_manager.cc:1916] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260616 01:58:03.296248 3958 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a719e057ca84bbb9dcb3a8320777481 with delete_type TABLET_DATA_DELETED (replaced by tablet de807bc8ba2c4132909082130958776e) from {username='slave'} at 127.0.0.1:34702
I20260616 01:58:03.300592 4248 tablet_service.cc:1511] Processing CreateTablet for tablet de807bc8ba2c4132909082130958776e (DEFAULT_TABLE table=default.table_59ad6b6ba51b44858e1003fece56722a [id=6035dfb0b8d94f3c981a30d83469e645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:03.301725 4248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de807bc8ba2c4132909082130958776e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:03.307629 4601 tablet_bootstrap.cc:492] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e: Bootstrap starting.
I20260616 01:58:03.311160 3958 tablet_service.cc:1511] Processing CreateTablet for tablet e622147476e046eda3b9a698d57abc5c (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:03.312286 3958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e622147476e046eda3b9a698d57abc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:03.312237 4105 tablet_service.cc:1511] Processing CreateTablet for tablet e622147476e046eda3b9a698d57abc5c (DEFAULT_TABLE table=default.table_3692a87561694c028a780ad53630b478 [id=c198a7e9a1424db1b682885a74a7f019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260616 01:58:03.313410 4105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e622147476e046eda3b9a698d57abc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:03.313505 4601 tablet_bootstrap.cc:654] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e: Neither blocks nor log segments found. Creating new log.
@ 0x7f05cb8cd61f clone at ??:0
I20260616 01:58:03.322618 4596 ts_tablet_manager.cc:1929] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260616 01:58:03.323091 4596 log.cc:1199] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-2/wal/wals/3a719e057ca84bbb9dcb3a8320777481
I20260616 01:58:03.324002 4596 ts_tablet_manager.cc:1950] T 3a719e057ca84bbb9dcb3a8320777481 P 3254c8a756fb47a78c049537dbe5e110: Deleting consensus metadata
I20260616 01:58:03.325903 4603 tablet_bootstrap.cc:492] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d: No bootstrap required, opened a new log
I20260616 01:58:03.326375 4603 ts_tablet_manager.cc:1403] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d: Time spent bootstrapping tablet: real 0.186s user 0.017s sys 0.038s
I20260616 01:58:03.331212 4604 tablet_bootstrap.cc:492] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110: No bootstrap required, opened a new log
I20260616 01:58:03.331651 4604 ts_tablet_manager.cc:1403] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110: Time spent bootstrapping tablet: real 0.185s user 0.019s sys 0.035s
I20260616 01:58:03.331977 4595 ts_tablet_manager.cc:1929] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260616 01:58:03.332355 4595 log.cc:1199] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-1/wal/wals/cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:03.332705 4601 tablet_bootstrap.cc:492] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e: No bootstrap required, opened a new log
I20260616 01:58:03.333106 4601 ts_tablet_manager.cc:1403] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.009s
I20260616 01:58:03.335626 4601 raft_consensus.cc:359] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.336300 4601 raft_consensus.cc:385] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.336575 4601 raft_consensus.cc:740] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.336755 4595 ts_tablet_manager.cc:1950] T cf37e86d57d346dfae2e866bed3ef847 P f4d208b546004421a8157a05464cbe7e: Deleting consensus metadata
I20260616 01:58:03.336485 4604 raft_consensus.cc:359] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:03.338289 4601 consensus_queue.cc:260] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.339838 4603 raft_consensus.cc:359] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.340567 4603 raft_consensus.cc:385] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.340901 4603 raft_consensus.cc:740] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.341758 4604 raft_consensus.cc:385] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.342067 4604 raft_consensus.cc:740] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.342931 4604 consensus_queue.cc:260] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:03.349639 4601 ts_tablet_manager.cc:1434] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260616 01:58:03.351619 4604 ts_tablet_manager.cc:1434] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20260616 01:58:03.344327 4603 consensus_queue.cc:260] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.352327 4593 ts_tablet_manager.cc:1929] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260616 01:58:03.352728 4593 log.cc:1199] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: Deleting WAL directory at /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/ts-0/wal/wals/cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:03.353457 4593 ts_tablet_manager.cc:1950] T cf37e86d57d346dfae2e866bed3ef847 P dc87c967de1144a6986d9ff87150d21d: Deleting consensus metadata
I20260616 01:58:03.353571 4603 ts_tablet_manager.cc:1434] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d: Time spent starting tablet: real 0.027s user 0.006s sys 0.002s
I20260616 01:58:03.354825 4604 tablet_bootstrap.cc:492] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110: Bootstrap starting.
I20260616 01:58:03.356648 4603 tablet_bootstrap.cc:492] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d: Bootstrap starting.
I20260616 01:58:03.364387 4603 tablet_bootstrap.cc:654] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.364790 4572 raft_consensus.cc:493] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:03.365314 4572 raft_consensus.cc:515] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.367498 4572 leader_election.cc:290] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:58:03.369592 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a2585bbddfd44aa95fdde7b0b52e0fd" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e" is_pre_election: true
I20260616 01:58:03.370210 4126 raft_consensus.cc:2468] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3254c8a756fb47a78c049537dbe5e110 in term 0.
I20260616 01:58:03.371220 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a2585bbddfd44aa95fdde7b0b52e0fd" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d" is_pre_election: true
I20260616 01:58:03.371784 3979 raft_consensus.cc:2468] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3254c8a756fb47a78c049537dbe5e110 in term 0.
I20260616 01:58:03.373023 4203 leader_election.cc:304] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:58:03.374369 4572 raft_consensus.cc:2804] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:58:03.374683 4572 raft_consensus.cc:493] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:58:03.374982 4572 raft_consensus.cc:3060] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:58:03.378573 4604 tablet_bootstrap.cc:654] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.385914 4601 tablet_bootstrap.cc:492] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e: Bootstrap starting.
I20260616 01:58:03.390285 4572 raft_consensus.cc:515] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.393330 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a2585bbddfd44aa95fdde7b0b52e0fd" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d"
I20260616 01:58:03.393853 3979 raft_consensus.cc:3060] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:58:03.394068 4604 tablet_bootstrap.cc:492] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110: No bootstrap required, opened a new log
I20260616 01:58:03.394449 4604 ts_tablet_manager.cc:1403] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110: Time spent bootstrapping tablet: real 0.040s user 0.016s sys 0.000s
I20260616 01:58:03.397081 4604 raft_consensus.cc:359] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.397778 4604 raft_consensus.cc:385] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.398046 4604 raft_consensus.cc:740] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.398715 4604 consensus_queue.cc:260] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.400699 3979 raft_consensus.cc:2468] T 1a2585bbddfd44aa95fdde7b0b52e0fd P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3254c8a756fb47a78c049537dbe5e110 in term 1.
I20260616 01:58:03.401952 4202 leader_election.cc:304] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110, dc87c967de1144a6986d9ff87150d21d; no voters:
I20260616 01:58:03.411253 4620 raft_consensus.cc:2804] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Leader election won for term 1
I20260616 01:58:03.411652 4620 raft_consensus.cc:697] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [term 1 LEADER]: Becoming Leader. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Running, Role: LEADER
I20260616 01:58:03.416404 4601 tablet_bootstrap.cc:654] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.416827 4604 ts_tablet_manager.cc:1434] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110: Time spent starting tablet: real 0.022s user 0.009s sys 0.000s
I20260616 01:58:03.412418 4620 consensus_queue.cc:237] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.418834 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a2585bbddfd44aa95fdde7b0b52e0fd" candidate_uuid: "3254c8a756fb47a78c049537dbe5e110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e"
I20260616 01:58:03.419358 4126 raft_consensus.cc:3060] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:58:03.434299 4126 raft_consensus.cc:2468] T 1a2585bbddfd44aa95fdde7b0b52e0fd P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3254c8a756fb47a78c049537dbe5e110 in term 1.
I20260616 01:58:03.435611 4572 leader_election.cc:290] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 election: Requested vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:58:03.447791 4603 tablet_bootstrap.cc:492] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d: No bootstrap required, opened a new log
I20260616 01:58:03.448200 4603 ts_tablet_manager.cc:1403] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d: Time spent bootstrapping tablet: real 0.092s user 0.037s sys 0.048s
I20260616 01:58:03.459116 4601 tablet_bootstrap.cc:492] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e: No bootstrap required, opened a new log
I20260616 01:58:03.458951 4603 raft_consensus.cc:359] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.459537 4601 ts_tablet_manager.cc:1403] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.002s
I20260616 01:58:03.459600 4603 raft_consensus.cc:385] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.459883 4603 raft_consensus.cc:740] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.460569 4603 consensus_queue.cc:260] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.463570 4601 raft_consensus.cc:359] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.464671 4601 raft_consensus.cc:385] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.464953 4601 raft_consensus.cc:740] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.467595 4603 ts_tablet_manager.cc:1434] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d: Time spent starting tablet: real 0.019s user 0.005s sys 0.002s
I20260616 01:58:03.469651 4603 tablet_bootstrap.cc:492] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d: Bootstrap starting.
I20260616 01:58:03.475560 4603 tablet_bootstrap.cc:654] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d: Neither blocks nor log segments found. Creating new log.
I20260616 01:58:03.480763 4603 tablet_bootstrap.cc:492] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d: No bootstrap required, opened a new log
I20260616 01:58:03.481289 4603 ts_tablet_manager.cc:1403] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260616 01:58:03.484583 4603 raft_consensus.cc:359] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.485459 4603 raft_consensus.cc:385] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260616 01:58:03.465596 4601 consensus_queue.cc:260] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.487944 4603 raft_consensus.cc:740] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Initialized, Role: FOLLOWER
I20260616 01:58:03.488669 4603 consensus_queue.cc:260] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.490218 4601 ts_tablet_manager.cc:1434] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e: Time spent starting tablet: real 0.030s user 0.005s sys 0.003s
I20260616 01:58:03.490816 4603 ts_tablet_manager.cc:1434] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20260616 01:58:03.605082 4620 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:03.605549 4620 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:03.607370 4620 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:03.609812 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:03.610529 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:03.610890 4202 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:03.611828 4620 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 01:58:03.666785 4205 connection.cc:573] server connection from 127.0.0.1:55854 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:58:03.666790 3914 connection.cc:573] server connection from 127.0.0.1:34702 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:58:03.667593 4059 connection.cc:573] client connection to 127.3.56.62:35519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:58:03.667769 4058 connection.cc:573] server connection from 127.0.0.1:35402 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:58:03.668447 4326 connection.cc:573] client connection to 127.3.56.62:35519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:58:03.669590 3912 connection.cc:573] client connection to 127.3.56.62:35519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:58:03.670363 4326 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.3.56.62:35519)
W20260616 01:58:03.671471 4326 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.56.62:35519)
W20260616 01:58:03.672943 4326 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.56.62:35519)
W20260616 01:58:03.679462 4203 connection.cc:573] client connection to 127.3.56.62:35519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260616 01:58:03.680409 4326 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.56.62:35519)
I20260616 01:58:03.767266 4628 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f509c7b2b84e4bd1a6e09d901dd331b7)
I20260616 01:58:03.767688 4628 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:03.771513 4628 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:03.773592 4487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:03.778762 4487 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:03.779426 4489 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:03.779768 4489 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:03.780395 4628 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:03.816335 4606 raft_consensus.cc:493] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:03.816874 4606 raft_consensus.cc:515] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.818898 4606 leader_election.cc:290] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:58:03.819710 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de807bc8ba2c4132909082130958776e" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d" is_pre_election: true
I20260616 01:58:03.820060 4269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de807bc8ba2c4132909082130958776e" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110" is_pre_election: true
I20260616 01:58:03.820297 3979 raft_consensus.cc:2468] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 0.
I20260616 01:58:03.820654 4269 raft_consensus.cc:2468] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 0.
I20260616 01:58:03.821260 4058 leader_election.cc:304] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:58:03.822029 4606 raft_consensus.cc:2804] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:58:03.822326 4606 raft_consensus.cc:493] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:58:03.822605 4606 raft_consensus.cc:3060] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:58:03.828058 4606 raft_consensus.cc:515] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:03.830761 3979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de807bc8ba2c4132909082130958776e" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc87c967de1144a6986d9ff87150d21d"
I20260616 01:58:03.831248 3979 raft_consensus.cc:3060] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:58:03.831337 4269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de807bc8ba2c4132909082130958776e" candidate_uuid: "f4d208b546004421a8157a05464cbe7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254c8a756fb47a78c049537dbe5e110"
I20260616 01:58:03.831861 4269 raft_consensus.cc:3060] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:58:03.832240 4606 leader_election.cc:290] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 election: Requested vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:58:03.836364 4269 raft_consensus.cc:2468] T de807bc8ba2c4132909082130958776e P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 1.
I20260616 01:58:03.837100 3979 raft_consensus.cc:2468] T de807bc8ba2c4132909082130958776e P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d208b546004421a8157a05464cbe7e in term 1.
I20260616 01:58:03.837361 4060 leader_election.cc:304] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110, f4d208b546004421a8157a05464cbe7e; no voters:
I20260616 01:58:03.838100 4606 raft_consensus.cc:2804] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader election won for term 1
I20260616 01:58:03.839079 4606 raft_consensus.cc:697] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [term 1 LEADER]: Becoming Leader. State: Replica: f4d208b546004421a8157a05464cbe7e, State: Running, Role: LEADER
I20260616 01:58:03.840855 4606 consensus_queue.cc:237] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:04.039995 4635 raft_consensus.cc:493] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:04.040469 4635 raft_consensus.cc:515] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:04.042876 4635 leader_election.cc:290] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:58:04.043296 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e622147476e046eda3b9a698d57abc5c" candidate_uuid: "dc87c967de1144a6986d9ff87150d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e" is_pre_election: true
I20260616 01:58:04.043948 4126 raft_consensus.cc:2468] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc87c967de1144a6986d9ff87150d21d in term 0.
W20260616 01:58:04.044281 3913 leader_election.cc:343] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531): Not found: Tablet not found: e622147476e046eda3b9a698d57abc5c
I20260616 01:58:04.045706 3912 leader_election.cc:304] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters: 3254c8a756fb47a78c049537dbe5e110
I20260616 01:58:04.046474 4635 raft_consensus.cc:2804] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260616 01:58:04.046833 4635 raft_consensus.cc:493] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260616 01:58:04.047101 4635 raft_consensus.cc:3060] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 0 FOLLOWER]: Advancing to term 1
I20260616 01:58:04.051328 4635 raft_consensus.cc:515] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:04.052820 4635 leader_election.cc:290] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 election: Requested vote from peers f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579), 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:58:04.053571 4126 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e622147476e046eda3b9a698d57abc5c" candidate_uuid: "dc87c967de1144a6986d9ff87150d21d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d208b546004421a8157a05464cbe7e"
I20260616 01:58:04.054061 4126 raft_consensus.cc:3060] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e [term 0 FOLLOWER]: Advancing to term 1
W20260616 01:58:04.054920 3913 leader_election.cc:343] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531): Not found: Tablet not found: e622147476e046eda3b9a698d57abc5c
I20260616 01:58:04.059301 4126 raft_consensus.cc:2468] T e622147476e046eda3b9a698d57abc5c P f4d208b546004421a8157a05464cbe7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc87c967de1144a6986d9ff87150d21d in term 1.
I20260616 01:58:04.060215 3912 leader_election.cc:304] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e; no voters: 3254c8a756fb47a78c049537dbe5e110
I20260616 01:58:04.061167 4635 raft_consensus.cc:2804] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 1 FOLLOWER]: Leader election won for term 1
I20260616 01:58:04.062237 4635 raft_consensus.cc:697] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [term 1 LEADER]: Becoming Leader. State: Replica: dc87c967de1144a6986d9ff87150d21d, State: Running, Role: LEADER
I20260616 01:58:04.063107 4635 consensus_queue.cc:237] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:04.360427 4638 consensus_queue.cc:1048] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260616 01:58:04.375571 4638 consensus_queue.cc:1048] T 1a2585bbddfd44aa95fdde7b0b52e0fd P 3254c8a756fb47a78c049537dbe5e110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260616 01:58:04.437000 4649 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f509c7b2b84e4bd1a6e09d901dd331b7)
I20260616 01:58:04.437654 4649 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:04.439997 4649 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:04.450996 4489 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:58:04.451666 4487 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 01:58:04.451984 4487 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:04.452559 4649 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:04.470165 3296 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu with pid 4428
W20260616 01:58:04.504784 4327 connection.cc:573] client connection to 127.3.56.60:37239 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260616 01:58:04.869916 4635 consensus_queue.cc:1048] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260616 01:58:04.880335 3913 consensus_peers.cc:597] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d -> Peer 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531): Couldn't send request to peer 3254c8a756fb47a78c049537dbe5e110. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e622147476e046eda3b9a698d57abc5c. This is attempt 1: this message will repeat every 5th retry.
I20260616 01:58:05.005707 4639 consensus_queue.cc:1048] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260616 01:58:05.023061 4662 consensus_queue.cc:1048] T de807bc8ba2c4132909082130958776e P f4d208b546004421a8157a05464cbe7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260616 01:58:05.211627 3296 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
/tmp/dist-test-taskkSI17J/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.60:37239
--webserver_interface=127.3.56.60
--webserver_port=34221
--builtin_ntp_servers=127.3.56.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260616 01:58:05.545251 4666 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260616 01:58:05.546093 4666 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260616 01:58:05.546808 4666 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260616 01:58:05.582628 4666 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260616 01:58:05.582971 4666 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260616 01:58:05.583240 4666 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260616 01:58:05.583506 4666 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260616 01:58:05.623196 4666 master_runner.cc:387] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.56.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:32849
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.56.60:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.56.60
--webserver_port=34221
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3396d19fb3cae30ffddb1de1e8c4632645aea710
build type FASTDEBUG
built by None at 16 Jun 2026 01:43:22 UTC on e7f111948823
build id 12415
TSAN enabled
I20260616 01:58:05.624523 4666 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260616 01:58:05.626145 4666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260616 01:58:05.635815 4677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:58:05.650408 4683 ts_tablet_manager.cc:933] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: Initiating tablet copy from peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353)
I20260616 01:58:05.653316 4683 tablet_copy_client.cc:323] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: tablet copy: Beginning tablet copy session from remote peer at address 127.3.56.1:34353
I20260616 01:58:05.655632 3999 tablet_copy_service.cc:140] P dc87c967de1144a6986d9ff87150d21d: Received BeginTabletCopySession request for tablet e622147476e046eda3b9a698d57abc5c from peer 3254c8a756fb47a78c049537dbe5e110 ({username='slave'} at 127.3.56.3:32891)
I20260616 01:58:05.656311 3999 tablet_copy_service.cc:161] P dc87c967de1144a6986d9ff87150d21d: Beginning new tablet copy session on tablet e622147476e046eda3b9a698d57abc5c from peer 3254c8a756fb47a78c049537dbe5e110 at {username='slave'} at 127.3.56.3:32891: session id = 3254c8a756fb47a78c049537dbe5e110-e622147476e046eda3b9a698d57abc5c
I20260616 01:58:05.671902 3999 tablet_copy_source_session.cc:215] T e622147476e046eda3b9a698d57abc5c P dc87c967de1144a6986d9ff87150d21d: Tablet Copy: opened 0 blocks and 1 log segments
I20260616 01:58:05.678961 4683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e622147476e046eda3b9a698d57abc5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260616 01:58:05.700650 4683 tablet_copy_client.cc:806] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: tablet copy: Starting download of 0 data blocks...
I20260616 01:58:05.701545 4683 tablet_copy_client.cc:670] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: tablet copy: Starting download of 1 WAL segments...
I20260616 01:58:05.706728 4683 tablet_copy_client.cc:538] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260616 01:58:05.718056 4683 tablet_bootstrap.cc:492] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: Bootstrap starting.
I20260616 01:58:05.743157 4683 tablet_bootstrap.cc:492] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260616 01:58:05.744510 4683 tablet_bootstrap.cc:492] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: Bootstrap complete.
I20260616 01:58:05.745357 4683 ts_tablet_manager.cc:1403] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.007s
I20260616 01:58:05.752667 4683 raft_consensus.cc:359] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:05.753787 4683 raft_consensus.cc:740] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254c8a756fb47a78c049537dbe5e110, State: Initialized, Role: FOLLOWER
I20260616 01:58:05.754884 4683 consensus_queue.cc:260] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } } peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } }
I20260616 01:58:05.758634 4683 ts_tablet_manager.cc:1434] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260616 01:58:05.760401 3999 tablet_copy_service.cc:342] P dc87c967de1144a6986d9ff87150d21d: Request end of tablet copy session 3254c8a756fb47a78c049537dbe5e110-e622147476e046eda3b9a698d57abc5c received from {username='slave'} at 127.3.56.3:32891
I20260616 01:58:05.760859 3999 tablet_copy_service.cc:434] P dc87c967de1144a6986d9ff87150d21d: ending tablet copy session 3254c8a756fb47a78c049537dbe5e110-e622147476e046eda3b9a698d57abc5c on tablet e622147476e046eda3b9a698d57abc5c with peer 3254c8a756fb47a78c049537dbe5e110
I20260616 01:58:06.724674 4269 raft_consensus.cc:1217] T e622147476e046eda3b9a698d57abc5c P 3254c8a756fb47a78c049537dbe5e110 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.1] Dedup: 1.1->[]
W20260616 01:58:05.637795 4678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260616 01:58:07.000986 4680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260616 01:58:07.005134 4666 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260616 01:58:07.005699 4679 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1367 milliseconds
I20260616 01:58:07.007344 4666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260616 01:58:07.013329 4666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260616 01:58:07.021317 4666 hybrid_clock.cc:648] HybridClock initialized: now 1781575087021227 us; error 73 us; skew 500 ppm
I20260616 01:58:07.022533 4666 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260616 01:58:07.030318 4666 webserver.cc:533] Webserver started at http://127.3.56.60:34221/ using document root <none> and password file <none>
I20260616 01:58:07.031239 4666 fs_manager.cc:362] Metadata directory not provided
I20260616 01:58:07.031476 4666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260616 01:58:07.040230 4666 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260616 01:58:07.045346 4706 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260616 01:58:07.046663 4666 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260616 01:58:07.047076 4666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
uuid: "673bf665c7d743579440c02560de26b9"
format_stamp: "Formatted at 2026-06-16 01:57:48 on dist-test-slave-s7k7"
I20260616 01:58:07.050086 4666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260616 01:58:07.105559 4666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260616 01:58:07.107015 4666 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260616 01:58:07.107426 4666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260616 01:58:07.178797 4666 rpc_server.cc:307] RPC server started. Bound to: 127.3.56.60:37239
I20260616 01:58:07.178886 4757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.56.60:37239 every 8 connection(s)
I20260616 01:58:07.181593 4666 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkSI17J/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1781575047418089-3296-0/minicluster-data/master-2/data/info.pb
I20260616 01:58:07.182358 3296 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkSI17J/build/tsan/bin/kudu as pid 4666
I20260616 01:58:07.193562 4758 sys_catalog.cc:263] Verifying existing consensus state
I20260616 01:58:07.199894 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Bootstrap starting.
I20260616 01:58:07.217797 4758 log.cc:826] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Log is configured to *not* fsync() on all Append() calls
I20260616 01:58:07.286196 4024 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:58:07.286394 4315 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:58:07.287812 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=13 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260616 01:58:07.288854 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: Bootstrap complete.
I20260616 01:58:07.295214 4171 heartbeater.cc:344] Connected to a master server at 127.3.56.60:37239
I20260616 01:58:07.310591 4758 raft_consensus.cc:359] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:07.312489 4758 raft_consensus.cc:740] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673bf665c7d743579440c02560de26b9, State: Initialized, Role: FOLLOWER
I20260616 01:58:07.313102 4758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:07.315279 4769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } } }
I20260616 01:58:07.315892 4769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260616 01:58:07.316376 4758 sys_catalog.cc:565] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260616 01:58:07.336114 4783 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260616 01:58:07.342074 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:07.361390 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:07.604215 4768 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:07.604751 4768 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:07.606786 4768 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:07.608174 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:07.608629 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:07.608948 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:07.609555 4768 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:58:07.797979 4722 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" instance_seqno: 1781575072950480) as {username='slave'} at 127.3.56.1:40715; Asking this server to re-register.
I20260616 01:58:07.797986 4723 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" instance_seqno: 1781575077465778) as {username='slave'} at 127.3.56.3:38403; Asking this server to re-register.
I20260616 01:58:07.799592 4024 heartbeater.cc:461] Registering TS with master...
I20260616 01:58:07.799572 4315 heartbeater.cc:461] Registering TS with master...
I20260616 01:58:07.803174 4721 ts_manager.cc:194] Registered new tserver with Master: dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353)
I20260616 01:58:07.803313 4723 ts_manager.cc:194] Registered new tserver with Master: 3254c8a756fb47a78c049537dbe5e110 (127.3.56.3:41531)
I20260616 01:58:07.803166 4722 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f4d208b546004421a8157a05464cbe7e" instance_seqno: 1781575075463092) as {username='slave'} at 127.3.56.2:39393; Asking this server to re-register.
I20260616 01:58:07.804683 4171 heartbeater.cc:461] Registering TS with master...
I20260616 01:58:07.807436 4723 ts_manager.cc:194] Registered new tserver with Master: f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
I20260616 01:58:07.920042 4788 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:07.920533 4788 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:07.922931 4788 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:07.926226 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:07.929831 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:58:07.930393 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 01:58:07.930688 4708 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:07.931277 4788 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:08.343782 4783 catalog_manager.cc:1294] Loaded cluster ID: 640ff0fe2fa14eee9f5b81f68327cf28
I20260616 01:58:08.344009 4783 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: loading cluster ID for follower catalog manager: success
I20260616 01:58:08.349771 4783 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: acquiring CA information for follower catalog manager: success
I20260616 01:58:08.354024 4783 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260616 01:58:08.582746 4798 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:08.583225 4798 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:08.584829 4798 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:08.591086 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:08.591601 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:08.591923 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:08.592535 4798 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:09.415973 4804 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:09.416499 4804 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:09.418675 4804 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:09.425601 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:09.426294 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:09.426640 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:09.427191 4804 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:09.529855 3296 master-stress-test.cc:579] Killing test thread 1/10
I20260616 01:58:10.505461 4815 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:10.505913 4815 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:10.507843 4815 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:10.513897 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:10.514513 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:10.514815 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:10.515357 4815 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:11.372635 4824 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:11.373039 4824 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:11.374577 4824 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:11.379288 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:11.379729 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:11.380059 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:11.380656 4824 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:12.364081 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:12.364472 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:12.460533 4838 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:12.460950 4838 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:12.462474 4838 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:12.466755 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:12.467350 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:12.467756 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:12.468453 4838 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:14.230242 4854 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:14.230723 4854 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:14.232259 4854 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:14.233887 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:14.234256 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:14.234549 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:14.235026 4854 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:58:14.972272 4870 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:14.972720 4870 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:14.974274 4870 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:14.976408 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260616 01:58:14.979086 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:14.979743 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:14.980051 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:14.980611 4870 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:17.367865 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:17.368405 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:18.314050 4918 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:18.314590 4918 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:18.316458 4918 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:18.322715 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:58:18.324434 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 01:58:18.324788 4708 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:18.325464 4918 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:21.913702 4976 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:21.914121 4976 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:21.915706 4976 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:21.917376 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260616 01:58:21.920441 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:21.920855 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:21.921155 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:21.921713 4976 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:22.370797 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:22.371184 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:23.627910 4995 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:23.628445 4995 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:23.630589 4995 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:23.632261 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:23.633107 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:23.633476 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:23.634002 4995 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:58:27.374680 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:27.375216 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:27.397712 5042 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:27.398255 5042 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:27.399875 5042 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:27.401356 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:27.401867 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:27.402123 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:27.402581 5042 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:58:31.937094 5115 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:31.937747 5115 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:31.939641 5115 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:31.941851 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 01:58:31.945300 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:31.945937 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:31.946205 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:31.946761 5115 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:32.377691 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:32.378054 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 01:58:35.301604 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2e80 after lost signal to thread 4182
W20260616 01:58:35.302492 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000be040 after lost signal to thread 4314
I20260616 01:58:37.081919 5190 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:37.082700 5190 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:37.084797 5190 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:37.086511 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:37.087114 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:37.087395 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:37.087965 5190 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:58:37.381466 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:37.381879 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:42.384433 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:42.384749 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:47.268617 5336 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:47.269284 5336 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:47.271637 5336 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:47.274317 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 01:58:47.277343 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:47.277915 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:47.278184 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:47.278734 5336 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:47.387153 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:47.387579 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:50.609220 5390 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:50.609684 5390 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:50.611554 5390 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:50.616478 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:50.617034 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:50.617496 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:50.618090 5390 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:51.280306 5401 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:51.280869 5401 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:51.283036 5401 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:51.284644 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:51.285130 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:51.285475 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:51.285995 5401 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 01:58:51.394078 4311 debug-util.cc:398] Leaking SignalData structure 0x7b0800042780 after lost signal to thread 4182
W20260616 01:58:51.394907 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf340 after lost signal to thread 4314
I20260616 01:58:52.390424 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:52.390874 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:58:53.639911 5441 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:53.640347 5441 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:53.642189 5441 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:53.644349 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260616 01:58:53.648180 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:58:53.648809 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 01:58:53.649271 4708 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:53.650071 5441 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:55.919260 5474 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:55.919673 5474 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:58:55.921181 5474 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:58:55.922734 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:58:55.923139 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:58:55.923393 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:58:55.923969 5474 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:58:56.696687 5490 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:58:56.697180 5490 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:58:56.698881 5490 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:58:56.704502 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:56.705088 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:58:56.705459 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:58:56.706063 5490 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:58:57.393873 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:58:57.394336 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 01:58:58.080960 4316 heartbeater.cc:646] Failed to heartbeat to 127.3.56.61:35255 (131 consecutive failures): Network error: Failed to ping master at 127.3.56.61:35255: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:58:58.108086 4173 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:35519 (114 consecutive failures): Network error: Failed to ping master at 127.3.56.62:35519: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:58.563585 4030 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:35519 (114 consecutive failures): Network error: Failed to ping master at 127.3.56.62:35519: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:58:58.711015 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000c65c0 after lost signal to thread 4036
W20260616 01:58:58.712046 4167 debug-util.cc:398] Leaking SignalData structure 0x7b0800040720 after lost signal to thread 4170
W20260616 01:58:59.796556 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000cbc20 after lost signal to thread 4182
W20260616 01:58:59.797472 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000415a0 after lost signal to thread 4314
I20260616 01:59:02.397020 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:02.397446 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:05.587265 5630 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:05.587733 5630 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:59:05.589478 5630 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:59:05.591142 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:59:05.591620 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:59:05.591923 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:59:05.592445 5630 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:59:07.400864 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:07.401338 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:10.272588 5704 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:10.273109 5704 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:59:10.275082 5704 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:59:10.277125 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260616 01:59:10.280287 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:59:10.280812 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 01:59:10.281142 4708 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:59:10.281796 5704 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:59:12.404139 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:12.404527 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:17.408155 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:17.408667 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 01:59:19.382777 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000db900 after lost signal to thread 4036
W20260616 01:59:19.383633 4167 debug-util.cc:398] Leaking SignalData structure 0x7b0800041700 after lost signal to thread 4170
I20260616 01:59:22.411674 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:22.412114 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:24.246194 5924 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:24.246636 5924 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:59:24.248225 5924 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:59:24.250120 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 01:59:24.253266 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:59:24.253777 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:59:24.254055 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:59:24.254607 5924 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:59:24.491369 5913 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:24.491971 5913 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:59:24.494347 5913 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:59:24.496003 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:59:24.496529 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:59:24.496819 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:59:24.497411 5913 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 01:59:27.166921 4020 debug-util.cc:398] Leaking SignalData structure 0x7b08000ccba0 after lost signal to thread 3861
W20260616 01:59:27.167822 4020 debug-util.cc:398] Leaking SignalData structure 0x7b08000cafa0 after lost signal to thread 4023
I20260616 01:59:27.419559 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:27.420068 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:31.599087 6018 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:31.599633 6018 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:59:31.601482 6018 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:59:31.603247 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:59:31.603763 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:59:31.604031 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:59:31.604559 6018 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 01:59:32.422993 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:32.423381 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:37.427466 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:37.428109 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:37.595026 6096 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:37.595574 6096 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:59:37.597433 6096 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:59:37.598824 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 01:59:37.601837 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:59:37.602358 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 01:59:37.602625 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:59:37.603161 6096 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 01:59:42.430770 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:42.431123 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:47.434429 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:47.434819 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:48.615614 6257 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:48.616214 6257 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 01:59:48.618507 6257 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 01:59:48.620173 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 01:59:48.620724 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 01:59:48.621037 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 01:59:48.621591 6257 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 01:59:50.778605 4020 debug-util.cc:398] Leaking SignalData structure 0x7b0800042a40 after lost signal to thread 3861
W20260616 01:59:50.780073 4020 debug-util.cc:398] Leaking SignalData structure 0x7b0800041320 after lost signal to thread 4023
I20260616 01:59:52.437383 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:52.437788 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 01:59:53.640484 6340 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 01:59:53.641067 6340 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 01:59:53.648074 6340 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 01:59:53.649494 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 01:59:53.654790 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:59:53.661743 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 01:59:53.662175 4708 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 01:59:53.662894 6340 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260616 01:59:54.677657 4020 debug-util.cc:398] Leaking SignalData structure 0x7b08000b91e0 after lost signal to thread 3861
W20260616 01:59:54.678524 4020 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4340 after lost signal to thread 4023
I20260616 01:59:57.441814 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 01:59:57.442413 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 01:59:58.497028 4316 heartbeater.cc:646] Failed to heartbeat to 127.3.56.61:35255 (251 consecutive failures): Network error: Failed to ping master at 127.3.56.61:35255: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 01:59:58.518149 4173 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:35519 (234 consecutive failures): Network error: Failed to ping master at 127.3.56.62:35519: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 01:59:58.964977 4030 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:35519 (234 consecutive failures): Network error: Failed to ping master at 127.3.56.62:35519: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 02:00:02.445154 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:02.445578 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:00:03.654101 4754 debug-util.cc:398] Leaking SignalData structure 0x7b08000700a0 after lost signal to thread 4669
W20260616 02:00:03.654662 4754 debug-util.cc:398] Leaking SignalData structure 0x7b0800078300 after lost signal to thread 4757
I20260616 02:00:04.173306 6486 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:04.173844 6486 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:00:04.175904 6486 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:00:04.177932 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:00:04.179359 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:00:04.179739 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:00:04.180544 6509 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 02:00:06.372246 4754 debug-util.cc:398] Leaking SignalData structure 0x7b08000782a0 after lost signal to thread 4669
W20260616 02:00:06.373108 4754 debug-util.cc:398] Leaking SignalData structure 0x7b08000707a0 after lost signal to thread 4757
I20260616 02:00:07.411675 4712 maintenance_manager.cc:419] P 673bf665c7d743579440c02560de26b9: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033360
I20260616 02:00:07.485047 4711 maintenance_manager.cc:643] P 673bf665c7d743579440c02560de26b9: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.072s user 0.025s sys 0.021s Metrics: {"bytes_written":9089,"cfile_init":1,"dirs.queue_time_us":785,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":922,"drs_written":1,"lbm_read_time_us":198,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1955,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":13,"thread_start_us":839,"threads_started":2,"wal-append.queue_time_us":647}
I20260616 02:00:07.487402 4712 maintenance_manager.cc:419] P 673bf665c7d743579440c02560de26b9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 840 bytes on disk
I20260616 02:00:07.488886 4711 maintenance_manager.cc:643] P 673bf665c7d743579440c02560de26b9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":236,"lbm_reads_lt_1ms":4}
I20260616 02:00:07.495823 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:07.496238 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:08.815104 6581 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:08.815615 6581 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:00:08.818084 6581 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:00:08.820670 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:00:08.823661 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:00:08.824165 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:00:08.824468 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:00:08.825111 6581 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:00:12.499056 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:12.499439 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:17.362370 6739 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:17.363076 6739 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:00:17.364945 6739 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:00:17.366318 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:00:17.366783 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:00:17.367085 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:00:17.367609 6739 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:00:17.502807 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:17.503271 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:20.140859 6804 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:20.141362 6804 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:00:20.142999 6804 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:00:20.145925 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:00:20.149919 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:00:20.150724 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:00:20.151116 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:00:20.151851 6804 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260616 02:00:21.128507 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000c52c0 after lost signal to thread 4182
W20260616 02:00:21.129375 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf240 after lost signal to thread 4314
I20260616 02:00:22.506266 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:22.506629 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:26.525789 6918 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:26.526296 6918 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:00:26.527983 6918 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:00:26.530757 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:00:26.533732 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 02:00:26.534134 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 02:00:26.534430 4708 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:00:26.534973 6918 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:00:27.509522 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:27.509900 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:32.512567 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:32.512893 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:37.243067 7096 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:37.243785 7096 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:00:37.246899 7096 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:00:37.248201 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:00:37.248767 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:00:37.249013 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:00:37.249532 7096 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:00:37.516371 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:37.516844 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:40.905613 7160 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:40.906109 7160 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:00:40.907817 7160 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:00:40.909762 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:00:40.912602 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:00:40.913275 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:00:40.913560 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:00:40.914108 7160 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:00:42.519816 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:42.520210 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:44.150065 7214 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:44.150552 7214 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:00:44.152459 7214 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:00:44.157699 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:00:44.158195 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:00:44.158560 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:00:44.159142 7214 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:00:45.396214 7229 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:45.396677 7229 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:00:45.398623 7229 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:00:45.399973 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:00:45.400432 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:00:45.400805 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:00:45.401511 7229 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:00:47.523334 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:47.523783 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:00:50.433110 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4780 after lost signal to thread 4036
W20260616 02:00:50.433983 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad4e0 after lost signal to thread 4170
I20260616 02:00:51.694125 7331 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:51.694626 7331 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:00:51.696404 7331 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:00:51.698946 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260616 02:00:51.702001 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:00:51.702616 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:00:51.702962 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:00:51.703649 7331 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:00:52.526197 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:52.526607 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:00:57.530895 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:00:57.531512 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:00:58.861277 4316 heartbeater.cc:646] Failed to heartbeat to 127.3.56.61:35255 (371 consecutive failures): Network error: Failed to ping master at 127.3.56.61:35255: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 02:00:58.913827 4173 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:35519 (354 consecutive failures): Network error: Failed to ping master at 127.3.56.62:35519: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 02:00:59.037492 7470 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:00:59.038028 7470 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:00:59.040120 7470 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:00:59.041656 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:00:59.042202 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:00:59.042469 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:00:59.042980 7470 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 02:00:59.376658 4025 heartbeater.cc:646] Failed to heartbeat to 127.3.56.61:35255 (373 consecutive failures): Network error: Failed to ping master at 127.3.56.61:35255: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:01:02.534343 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:02.534754 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:05.708261 7590 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:05.708649 7590 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:01:05.710436 7590 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:01:05.711925 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:01:05.712321 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:01:05.712677 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:01:05.713276 7590 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:01:06.236904 7602 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:06.237596 7602 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:01:06.239385 7602 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:01:06.241755 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:01:06.244801 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:01:06.245329 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:01:06.245644 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:01:06.246199 7602 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:01:07.537671 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:07.538100 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:01:07.820570 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000cbe60 after lost signal to thread 4182
W20260616 02:01:07.821457 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf920 after lost signal to thread 4314
W20260616 02:01:09.393646 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000d19c0 after lost signal to thread 4036
W20260616 02:01:09.394867 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3540 after lost signal to thread 4170
I20260616 02:01:10.554111 7654 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:10.554632 7654 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:01:10.556299 7654 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:01:10.557946 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:01:10.558439 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:01:10.558686 4202 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:01:10.559144 7654 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 02:01:11.741796 4020 debug-util.cc:398] Leaking SignalData structure 0x7b08000d18c0 after lost signal to thread 3861
W20260616 02:01:11.742645 4020 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfa60 after lost signal to thread 4023
I20260616 02:01:12.540855 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:12.541183 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:17.544927 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:17.545409 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:17.883069 7824 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:17.883576 7824 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:01:17.885272 7824 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:01:17.886698 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:01:17.889755 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:01:17.890276 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:01:17.890575 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:01:17.891124 7824 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:01:22.547904 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:22.548272 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:01:23.256582 4020 debug-util.cc:398] Leaking SignalData structure 0x7b0800040260 after lost signal to thread 3861
W20260616 02:01:23.257704 4020 debug-util.cc:398] Leaking SignalData structure 0x7b0800040e40 after lost signal to thread 4023
I20260616 02:01:23.537551 7919 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:23.538051 7919 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:01:23.540206 7919 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:01:23.542822 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:01:23.543442 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:01:23.543720 4202 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:01:23.544299 7919 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260616 02:01:27.475818 4754 debug-util.cc:398] Leaking SignalData structure 0x7b080007c140 after lost signal to thread 4669
W20260616 02:01:27.476387 4754 debug-util.cc:398] Leaking SignalData structure 0x7b08000702a0 after lost signal to thread 4757
I20260616 02:01:27.579309 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:27.579704 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:32.582556 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:32.583039 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:33.133613 8115 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:33.134150 8115 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
W20260616 02:01:33.138515 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260616 02:01:33.143911 8115 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:01:33.152773 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:01:33.153364 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:01:33.153636 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:01:33.154170 8115 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:01:34.699846 8128 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:34.700376 8128 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:01:34.702152 8128 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:01:34.703640 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:01:34.704139 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:01:34.704427 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:01:34.705029 8128 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:01:37.586716 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:37.587159 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:01:39.351496 4020 debug-util.cc:398] Leaking SignalData structure 0x7b0800042ea0 after lost signal to thread 3861
W20260616 02:01:39.352371 4020 debug-util.cc:398] Leaking SignalData structure 0x7b080006fd20 after lost signal to thread 4023
I20260616 02:01:42.589911 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:42.590283 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:43.359697 8302 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:43.360261 8302 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:01:43.362469 8302 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:01:43.365868 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:01:43.369470 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:01:43.370015 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:01:43.370321 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:01:43.370854 8302 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260616 02:01:46.875492 4754 debug-util.cc:398] Leaking SignalData structure 0x7b080007c2e0 after lost signal to thread 4669
W20260616 02:01:46.876463 4754 debug-util.cc:398] Leaking SignalData structure 0x7b0800070180 after lost signal to thread 4757
I20260616 02:01:47.633710 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:47.634130 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:48.687674 8394 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:48.688462 8394 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:01:48.690755 8394 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:01:48.692149 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:01:48.692683 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:01:48.692934 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:01:48.693459 8394 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:01:52.636826 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:52.637157 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:57.640673 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:01:57.641131 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:01:58.287308 8580 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:01:58.287840 8580 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:01:58.290222 8580 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:01:58.293459 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:01:58.296691 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 02:01:58.297276 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 02:01:58.297565 4708 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:01:58.298180 8580 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260616 02:01:59.254026 4316 heartbeater.cc:646] Failed to heartbeat to 127.3.56.61:35255 (491 consecutive failures): Network error: Failed to ping master at 127.3.56.61:35255: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 02:01:59.302229 4172 heartbeater.cc:646] Failed to heartbeat to 127.3.56.61:35255 (493 consecutive failures): Network error: Failed to ping master at 127.3.56.61:35255: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
W20260616 02:01:59.763820 4030 heartbeater.cc:646] Failed to heartbeat to 127.3.56.62:35519 (474 consecutive failures): Network error: Failed to ping master at 127.3.56.62:35519: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
I20260616 02:02:02.318696 8637 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:02.319152 8637 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:02:02.320694 8637 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:02:02.322284 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:02:02.322850 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:02:02.323123 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:02:02.323632 8637 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:02:02.644115 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:02.644505 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:07.647884 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:07.648327 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:02:11.458747 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfaa0 after lost signal to thread 4182
W20260616 02:02:11.459681 4311 debug-util.cc:398] Leaking SignalData structure 0x7b0800040b80 after lost signal to thread 4314
I20260616 02:02:12.651247 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:12.651648 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:13.709393 8810 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:13.709896 8810 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:02:13.711804 8810 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:02:13.714039 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:02:13.716935 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:13.717468 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:02:13.717756 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:02:13.718324 8810 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:02:14.921497 8840 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:14.921938 8840 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:02:14.923856 8840 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:02:14.928114 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:14.928464 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:02:14.928737 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:02:14.929246 8840 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:02:17.654837 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:17.655292 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:19.332996 8910 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:19.333767 8910 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:02:19.336264 8910 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:02:19.337657 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:02:19.338112 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:02:19.338434 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:02:19.338954 8910 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:02:22.658031 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:22.658474 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:27.661741 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:27.662207 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:30.939213 9131 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:30.939671 9131 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:02:30.941704 9131 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:02:30.943666 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260616 02:02:30.947057 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:30.947626 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:02:30.947881 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:02:30.948376 9131 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:02:32.665514 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:32.665920 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:36.034776 9223 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:36.035247 9223 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:02:36.037104 9223 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:02:36.038748 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:02:36.039306 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:02:36.039652 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:02:36.040385 9223 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:02:37.669239 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:37.669795 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:02:41.002693 4311 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6e80 after lost signal to thread 4182
W20260616 02:02:41.003510 4311 debug-util.cc:398] Leaking SignalData structure 0x7b080006f920 after lost signal to thread 4314
I20260616 02:02:41.535041 9295 raft_consensus.cc:493] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:41.535560 9295 raft_consensus.cc:515] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3254c8a756fb47a78c049537dbe5e110" member_type: VOTER last_known_addr { host: "127.3.56.3" port: 41531 } } peers { permanent_uuid: "dc87c967de1144a6986d9ff87150d21d" member_type: VOTER last_known_addr { host: "127.3.56.1" port: 34353 } } peers { permanent_uuid: "f4d208b546004421a8157a05464cbe7e" member_type: VOTER last_known_addr { host: "127.3.56.2" port: 41579 } }
I20260616 02:02:41.537326 9295 leader_election.cc:290] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353), f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579)
W20260616 02:02:41.538802 4202 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc87c967de1144a6986d9ff87150d21d (127.3.56.1:34353): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
W20260616 02:02:41.539251 4203 leader_election.cc:343] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f4d208b546004421a8157a05464cbe7e (127.3.56.2:41579): Not found: Tablet not found: cf37e86d57d346dfae2e866bed3ef847
I20260616 02:02:41.539527 4203 leader_election.cc:304] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3254c8a756fb47a78c049537dbe5e110; no voters: dc87c967de1144a6986d9ff87150d21d, f4d208b546004421a8157a05464cbe7e
I20260616 02:02:41.539968 9295 raft_consensus.cc:2749] T cf37e86d57d346dfae2e866bed3ef847 P 3254c8a756fb47a78c049537dbe5e110 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260616 02:02:42.672504 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:42.672843 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:43.142038 9338 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:43.142522 9338 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:02:43.144093 9338 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:02:43.145798 4708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260616 02:02:43.149122 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:43.149802 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:02:43.150070 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:02:43.150657 9338 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:02:47.676982 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:47.677474 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260616 02:02:48.098801 9394 raft_consensus.cc:493] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260616 02:02:48.099332 9394 raft_consensus.cc:515] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f509c7b2b84e4bd1a6e09d901dd331b7" member_type: VOTER last_known_addr { host: "127.3.56.62" port: 35519 } } peers { permanent_uuid: "321530d69d794e8f94c5424224a44115" member_type: VOTER last_known_addr { host: "127.3.56.61" port: 35255 } } peers { permanent_uuid: "673bf665c7d743579440c02560de26b9" member_type: VOTER last_known_addr { host: "127.3.56.60" port: 37239 } }
I20260616 02:02:48.101014 9394 leader_election.cc:290] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519), 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255)
W20260616 02:02:48.105604 4708 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f509c7b2b84e4bd1a6e09d901dd331b7 (127.3.56.62:35519): Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:48.106675 4710 leader_election.cc:336] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 321530d69d794e8f94c5424224a44115 (127.3.56.61:35255): Network error: Client connection negotiation failed: client connection to 127.3.56.61:35255: connect: Connection refused (error 111)
I20260616 02:02:48.106909 4710 leader_election.cc:304] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673bf665c7d743579440c02560de26b9; no voters: 321530d69d794e8f94c5424224a44115, f509c7b2b84e4bd1a6e09d901dd331b7
I20260616 02:02:48.107381 9394 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 673bf665c7d743579440c02560de26b9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260616 02:02:52.680244 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:52.680639 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:02:52.684108 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2e80 after lost signal to thread 4036
W20260616 02:02:52.684943 4167 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9a80 after lost signal to thread 4170
I20260616 02:02:57.684132 4666 master_runner.cc:187] Error getting master registration for 127.3.56.62:35519: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111),
I20260616 02:02:57.684605 4666 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260616 02:02:58.863852 4326 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:58.864498 4326 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:58.864897 4326 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
W20260616 02:02:58.865181 4326 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
F20260616 02:02:58.876788 4335 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_5aeb666f1fb64981ba79c4f3f4e2b82f on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260616 02:02:58.881412 4337 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1781575378 (unix time) try "date -d @1781575378" if you are using GNU date ***
W20260616 02:02:58.883960 4326 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.56.62:35519)
W20260616 02:02:58.884673 4326 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.3.56.62:35519)
W20260616 02:02:58.892441 4326 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.3.56.62:35519)
PC: @ 0x0 (unknown)
F20260616 02:02:58.881412 4337 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)F20260616 02:02:58.898283 4338 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.3.56.62:35519,127.3.56.61:35255,127.3.56.60:37239, num_attempts: 762) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.3.56.62:35519: connect: Connection refused (error 111)
*** SIGABRT (@0x3e800000ce0) received by PID 3296 (TID 0x7fc603dfd700) from PID 3296; stack trace: ***
@ 0x2de5f0 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e0ad3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
*** Check failure stack trace: ***
@ 0x7fc6166fe980 (unknown) at ??:0
@ 0x7fc6118bee87 gsignal at ??:0
@ 0x7fc6118c07f1 abort at ??:0
@ 0x2e45e6 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
==================
WARNING: ThreadSanitizer: data race (pid=3296)
Atomic write of size 8 at 0x7fc614b828b8 by thread T17:
#0 __tsan_atomic64_compare_exchange_val /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp:850 (master-stress-test+0x3385d8)
#1 unsigned long* google::glog_internal_namespace_::sync_val_compare_and_swap<unsigned long*>(unsigned long**, unsigned long*, unsigned long*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/utilities.h:172:10 (libglog.so.1+0x2b285)
#2 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/signalhandler.cc:296:7 (libglog.so.1+0x2b285)
#3 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1911 (master-stress-test+0x2de5ef)
#4 google::LogMessage::Fail() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1946:3 (libglog.so.1+0x18da3)
#5 google::LogMessage::SendToLog() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1920:5 (libglog.so.1+0x18da3)
#6 google::LogMessage::Flush() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1777:5 (libglog.so.1+0x1994f)
#7 google::LogMessageFatal::~LogMessageFatal() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:2557:5 (libglog.so.1+0x1ea8a)
#8 kudu::MasterStressTest::AlterTableThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:330:7 (master-stress-test+0x360d15)
#9 decltype(*(std::__1::forward<kudu::MasterStressTest_Test_Test*>(fp0)).*fp()) std::__1::__invoke<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846:1 (master-stress-test+0x38e87d)
#10 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>&, std::__1::__tuple_indices<2ul>) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:280:5 (master-stress-test+0x38e730)
#11 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*> >(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:291:5 (master-stress-test+0x38de7a)
Previous read of size 8 at 0x7fc614b828b8 by thread T16:
#0 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/signalhandler.cc:302:38 (libglog.so.1+0x2ca90)
#1 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1911 (master-stress-test+0x2de5ef)
#2 google::LogMessage::Fail() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1946:3 (libglog.so.1+0x18da3)
#3 google::LogMessage::SendToLog() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1920:5 (libglog.so.1+0x18da3)
#4 google::LogMessage::Flush() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1777:5 (libglog.so.1+0x1994f)
#5 google::LogMessageFatal::~LogMessageFatal() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:2557:5 (libglog.so.1+0x1ea8a)
#6 kudu::MasterStressTest::AlterTableThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:330:7 (master-stress-test+0x360d15)
#7 decltype(*(std::__1::forward<kudu::MasterStressTest_Test_Test*>(fp0)).*fp()) std::__1::__invoke<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846:1 (master-stress-test+0x38e87d)
#8 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>&, std::__1::__tuple_indices<2ul>) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:280:5 (master-stress-test+0x38e730)
#9 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*> >(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:291:5 (master-stress-test+0x38de7a)
Location is global 'google::(anonymous namespace)::g_entered_thread_id_pointer' of size 8 at 0x7fc614b828b8 (libglog.so.1+0x0000002448b8)
Thread T17 (tid=4338, running) created by main thread at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (master-stress-test+0x2e1f65)
#1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__threading_support:504:10 (master-stress-test+0x38de0b)
#2 std::__1::thread::thread<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:307:16 (master-stress-test+0x38db67)
#3 void std::__1::allocator<std::__1::thread>::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1809:31 (master-stress-test+0x38da6c)
#4 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::__construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1687:21 (master-stress-test+0x38da03)
#5 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1538:14 (master-stress-test+0x38d943)
#6 void std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::__construct_one_at_end<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:926:5 (master-stress-test+0x38d64e)
#7 std::__1::thread& std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::emplace_back<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:1687:9 (master-stress-test+0x36017a)
#8 kudu::MasterStressTest_Test_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:562:13 (master-stress-test+0x35f066)
#9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#11 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#12 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#13 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#17 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#18 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (master-stress-test+0x39106b)
#19 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:115:10 (master-stress-test+0x38fc7c)
Thread T16 (tid=4337, running) created by main thread at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (master-stress-test+0x2e1f65)
#1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__threading_support:504:10 (master-stress-test+0x38de0b)
#2 std::__1::thread::thread<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:307:16 (master-stress-test+0x38db67)
#3 void std::__1::allocator<std::__1::thread>::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1809:31 (master-stress-test+0x38da6c)
#4 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::__construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1687:21 (master-stress-test+0x38da03)
#5 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1538:14 (master-stress-test+0x38d943)
#6 void std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::__emplace_back_slow_path<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:1670:5 (master-stress-test+0x38d77e)
#7 std::__1::thread& std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::emplace_back<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:1690:9 (master-stress-test+0x36018a)
#8 kudu::MasterStressTest_Test_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:562:13 (master-stress-test+0x35f066)
#9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#11 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#12 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#13 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#17 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#18 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (master-stress-test+0x39106b)
#19 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:115:10 (master-stress-test+0x38fc7c)
SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/utilities.h:172:10 in unsigned long* google::glog_internal_namespace_::sync_val_compare_and_swap<unsigned long*>(unsigned long**, unsigned long*, unsigned long*)
==================
@ 0x7fc614956da4 google::LogMessage::SendToLog() at ??:0
@ 0x7fc614957950 google::LogMessage::Flush() at ??:0
@ 0x7fc61495ca8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x360724 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38e87e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38e731 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38de7b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2de4b0 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7fc6166f36db start_thread at ??:0
@ 0x7fc6119a161f clone at ??:0