Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
2026-11-18T08:54:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-18T08:54:07Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3eSafg/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3eSafg/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/security.properties
2026-11-18 08:54:14: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3eSafg/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task3eSafg/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20261118 08:54:21.594707 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--webserver_interface=127.0.72.190
--webserver_port=0
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 08:54:21.945640 572 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:21.945945 572 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:21.946048 572 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:21.954403 572 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:21.954492 572 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:21.954558 572 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:21.954618 572 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:21.965039 572 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.72.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:21.966674 572 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:21.968484 572 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:21.977042 577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:21.977046 578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:21.977902 580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:21.978412 572 server_base.cc:1061] running on GCE node
I20261118 08:54:21.979213 572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:21.980720 572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:21.981925 572 hybrid_clock.cc:648] HybridClock initialized: now 1794992061981836 us; error 80 us; skew 500 ppm
I20261118 08:54:21.982242 572 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:21.984352 572 webserver.cc:577] Webserver started at http://127.0.72.190:41231/ using document root <none> and password file <none>
I20261118 08:54:21.985061 572 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:21.985170 572 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:21.985992 572 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 08:54:21.988420 572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/instance:
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:21.989200 572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal/instance:
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:21.994464 572 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20261118 08:54:21.997740 586 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:21.999176 572 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20261118 08:54:21.999315 572 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:21.999569 572 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:22.022874 572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:22.023790 572 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:22.024034 572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:22.042620 572 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.190:42341
I20261118 08:54:22.042650 637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.190:42341 every 8 connection(s)
I20261118 08:54:22.044080 572 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
I20261118 08:54:22.048715 638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:22.051168 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 572
I20261118 08:54:22.051429 290 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal/instance
I20261118 08:54:22.053730 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--webserver_interface=127.0.72.189
--webserver_port=0
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 08:54:22.054512 638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20261118 08:54:22.070093 638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
W20261118 08:54:22.071106 588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.189:33939: connect: Connection refused (error 111)
W20261118 08:54:22.072595 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.189:33939: Network error: Client connection negotiation failed: client connection to 127.0.72.189:33939: connect: Connection refused (error 111)
I20261118 08:54:22.121907 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } attempt: 1
W20261118 08:54:22.123939 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.189:33939: Network error: Client connection negotiation failed: client connection to 127.0.72.189:33939: connect: Connection refused (error 111)
I20261118 08:54:22.192175 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } attempt: 2
W20261118 08:54:22.193980 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.189:33939: Network error: Client connection negotiation failed: client connection to 127.0.72.189:33939: connect: Connection refused (error 111)
I20261118 08:54:22.285224 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } attempt: 3
W20261118 08:54:22.287227 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.189:33939: Network error: Client connection negotiation failed: client connection to 127.0.72.189:33939: connect: Connection refused (error 111)
W20261118 08:54:22.416447 639 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:22.416770 639 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:22.416881 639 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:22.425249 639 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:22.425350 639 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:22.425828 639 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:22.425904 639 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:22.430450 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } attempt: 4
W20261118 08:54:22.432528 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.189:33939: Network error: Client connection negotiation failed: client connection to 127.0.72.189:33939: connect: Connection refused (error 111)
I20261118 08:54:22.436986 639 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.72.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:22.438602 639 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:22.440388 639 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:22.448717 650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:22.448680 647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:22.448748 648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:22.449851 639 server_base.cc:1061] running on GCE node
I20261118 08:54:22.450627 639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:22.452128 639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:22.453328 639 hybrid_clock.cc:648] HybridClock initialized: now 1794992062453275 us; error 41 us; skew 500 ppm
I20261118 08:54:22.453647 639 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:22.455760 639 webserver.cc:577] Webserver started at http://127.0.72.189:35929/ using document root <none> and password file <none>
I20261118 08:54:22.456449 639 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:22.456557 639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:22.456921 639 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 08:54:22.459403 639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/instance:
uuid: "857a577b88ba46cc846ecd1a25a722ca"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:22.460214 639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal/instance:
uuid: "857a577b88ba46cc846ecd1a25a722ca"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:22.466146 639 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20261118 08:54:22.469712 656 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:22.471447 639 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:22.471628 639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
uuid: "857a577b88ba46cc846ecd1a25a722ca"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:22.471835 639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:22.489454 639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:22.490382 639 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:22.490655 639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:22.511423 639 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.189:33939
I20261118 08:54:22.511436 707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.189:33939 every 8 connection(s)
I20261118 08:54:22.512964 639 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
I20261118 08:54:22.517221 708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:22.520051 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 639
I20261118 08:54:22.520226 290 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal/instance
I20261118 08:54:22.521242 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.188:41259
--webserver_interface=127.0.72.188
--webserver_port=0
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 08:54:22.523514 708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20261118 08:54:22.537590 708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
I20261118 08:54:22.546088 708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } has no permanent_uuid. Determining permanent_uuid...
W20261118 08:54:22.547586 657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:22.549081 708 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:22.598363 708 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 1
W20261118 08:54:22.600262 708 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:22.668488 708 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 2
W20261118 08:54:22.670584 708 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:22.731792 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } attempt: 5
I20261118 08:54:22.738198 638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } has no permanent_uuid. Determining permanent_uuid...
W20261118 08:54:22.740202 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:22.761814 708 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 3
W20261118 08:54:22.763927 708 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:22.791471 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 1
W20261118 08:54:22.793439 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:22.861689 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 2
W20261118 08:54:22.863569 638 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:22.869073 709 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:22.869367 709 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:22.869447 709 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:22.878365 709 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:22.878460 709 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:22.878511 709 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:22.878556 709 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:22.889273 709 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.188:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.72.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:22.890918 709 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:22.892711 709 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:22.901058 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:22.901014 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:22.901723 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:22.902417 709 server_base.cc:1061] running on GCE node
I20261118 08:54:22.903169 709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:22.904649 709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:22.905846 709 hybrid_clock.cc:648] HybridClock initialized: now 1794992062905775 us; error 64 us; skew 500 ppm
I20261118 08:54:22.906184 709 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:22.907152 708 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 4
I20261118 08:54:22.908560 709 webserver.cc:577] Webserver started at http://127.0.72.188:40343/ using document root <none> and password file <none>
W20261118 08:54:22.908911 708 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:22.909303 709 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:22.909402 709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:22.909785 709 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 08:54:22.912179 709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data/instance:
uuid: "6c7d9f52503f41e4bafe2821e6fd6af1"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:22.912923 709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/wal/instance:
uuid: "6c7d9f52503f41e4bafe2821e6fd6af1"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:22.918246 709 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20261118 08:54:22.921588 726 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:22.923233 709 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:22.923455 709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/wal
uuid: "6c7d9f52503f41e4bafe2821e6fd6af1"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:22.923692 709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:22.938583 709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:22.939836 709 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:22.940186 709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:22.962540 709 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.188:41259
I20261118 08:54:22.962553 777 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.188:41259 every 8 connection(s)
I20261118 08:54:22.964308 709 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/data/info.pb
I20261118 08:54:22.966774 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 709
I20261118 08:54:22.966959 290 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-2/wal/instance
I20261118 08:54:22.969305 778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:22.969786 638 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 3
I20261118 08:54:22.979097 778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20261118 08:54:22.999146 638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap starting.
I20261118 08:54:22.999230 778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } has no permanent_uuid. Determining permanent_uuid...
I20261118 08:54:23.003113 638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:23.004325 638 log.cc:826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:23.007369 778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } has no permanent_uuid. Determining permanent_uuid...
I20261118 08:54:23.008410 638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: No bootstrap required, opened a new log
I20261118 08:54:23.015676 638 raft_consensus.cc:360] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.015950 638 raft_consensus.cc:386] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:23.016064 638 raft_consensus.cc:748] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Initialized, Role: FOLLOWER
I20261118 08:54:23.016714 638 consensus_queue.cc:261] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.017597 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.017895 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:23.018667 778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: Bootstrap starting.
I20261118 08:54:23.018841 638 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:23.022773 778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:23.024398 778 log.cc:826] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: Log is configured to *not* fsync() on all Append() calls
W20261118 08:54:23.027036 592 tablet.cc:2441] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 08:54:23.027209 778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: No bootstrap required, opened a new log
I20261118 08:54:23.035250 778 raft_consensus.cc:360] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.035738 778 raft_consensus.cc:386] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:23.035916 778 raft_consensus.cc:748] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7d9f52503f41e4bafe2821e6fd6af1, State: Initialized, Role: FOLLOWER
I20261118 08:54:23.036852 778 consensus_queue.cc:261] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.043466 572 master_runner.cc:187] Error getting master registration for 127.0.72.189:33939: OK, instance_id { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" instance_seqno: 1794992062505753 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20261118 08:54:23.050841 803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.051213 803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:23.052399 778 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:23.054576 572 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:54:23.054319 801 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261118 08:54:23.054755 801 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261118 08:54:23.078428 786 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:23.078748 786 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.080739 786 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:23.082221 753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" is_pre_election: true
I20261118 08:54:23.082841 753 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 0.
I20261118 08:54:23.089746 587 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871; no voters:
I20261118 08:54:23.090960 786 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:23.091135 786 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:23.091241 786 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:23.094861 786 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.097041 753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1"
I20261118 08:54:23.097283 753 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:23.100651 753 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 1.
I20261118 08:54:23.101648 587 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871; no voters:
I20261118 08:54:23.109010 786 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 1 election: Requested vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:23.110280 786 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:23.110991 786 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 1 LEADER]: Becoming Leader. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Running, Role: LEADER
I20261118 08:54:23.113349 786 consensus_queue.cc:238] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.119354 813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 1 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.119717 813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:23.132269 821 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20261118 08:54:23.134711 819 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 08:54:23.137976 819 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 08:54:23.167618 803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 1 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.167964 803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:23.182780 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.183100 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:23.195333 803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.195598 803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:23.197758 709 master_runner.cc:187] Error getting master registration for 127.0.72.189:33939: OK, instance_id { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" instance_seqno: 1794992062505753 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20261118 08:54:23.208113 708 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } attempt: 5
I20261118 08:54:23.211732 709 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:23.221289 708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap starting.
I20261118 08:54:23.225664 708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:23.227665 708 log.cc:826] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Log is configured to *not* fsync() on all Append() calls
W20261118 08:54:23.231904 826 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261118 08:54:23.233011 826 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261118 08:54:23.237808 708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: No bootstrap required, opened a new log
I20261118 08:54:23.244191 819 catalog_manager.cc:1398] Generated new cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:23.244314 819 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 08:54:23.246789 708 raft_consensus.cc:360] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.247244 708 raft_consensus.cc:386] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:23.247762 708 raft_consensus.cc:748] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857a577b88ba46cc846ecd1a25a722ca, State: Initialized, Role: FOLLOWER
I20261118 08:54:23.248679 708 consensus_queue.cc:261] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.255983 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.258185 683 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
I20261118 08:54:23.258659 681 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:23.258157 708 sys_catalog.cc:565] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:23.261642 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:23.263510 821 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:23.263507 682 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca"
I20261118 08:54:23.267019 682 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 1.
I20261118 08:54:23.285120 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 1 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.285465 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:23.286087 837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.286484 837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:23.289351 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:23.289580 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:23.294399 819 catalog_manager.cc:1421] Generated new certificate authority record
I20261118 08:54:23.296689 819 catalog_manager.cc:1555] Loading token signing keys...
I20261118 08:54:23.323945 819 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Generated new TSK 0
I20261118 08:54:23.325229 819 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 08:54:23.326709 819 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 08:54:23.331487 819 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 08:54:23.356958 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.72.129:0
--local_ip_for_outbound_sockets=127.0.72.129
--webserver_interface=127.0.72.129
--webserver_port=0
--tserver_master_addrs=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20261118 08:54:23.394093 853 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:23.394212 853 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: loading cluster ID for follower catalog manager: success
I20261118 08:54:23.398656 853 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: acquiring CA information for follower catalog manager: success
I20261118 08:54:23.413873 853 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:23.845132 856 raft_consensus.cc:499] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:23.845541 856 raft_consensus.cc:521] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:23.847283 856 leader_election.cc:290] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939)
I20261118 08:54:23.847967 681 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
I20261118 08:54:23.847961 613 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78d4026bc230421ea10b8c68a9ef7871" is_pre_election: true
I20261118 08:54:23.848855 728 leader_election.cc:304] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c7d9f52503f41e4bafe2821e6fd6af1; no voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca
I20261118 08:54:23.849653 856 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:54:23.925673 855 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:23.926107 855 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:23.926268 855 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:23.939601 855 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:23.939872 855 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.72.129
I20261118 08:54:23.956419 855 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.72.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.72.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.72.129
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:23.958516 855 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:23.961050 855 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:23.972324 861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:23.975593 862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:23.983698 864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:23.983942 855 server_base.cc:1061] running on GCE node
I20261118 08:54:23.984627 855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:23.985702 855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:23.986915 855 hybrid_clock.cc:648] HybridClock initialized: now 1794992063986857 us; error 50 us; skew 500 ppm
I20261118 08:54:23.987224 855 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:23.990185 855 webserver.cc:577] Webserver started at http://127.0.72.129:36353/ using document root <none> and password file <none>
I20261118 08:54:23.990918 855 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:23.991019 855 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:23.991415 855 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 08:54:23.993945 855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data/instance:
uuid: "ad2789ff5b54489c84966cff78be4721"
format_stamp: "Formatted at 2026-11-18 08:54:23 on dist-test-slave-m8c8"
I20261118 08:54:23.994714 855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/instance:
uuid: "ad2789ff5b54489c84966cff78be4721"
format_stamp: "Formatted at 2026-11-18 08:54:23 on dist-test-slave-m8c8"
I20261118 08:54:24.000763 855 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20261118 08:54:24.004264 870 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:24.005724 855 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20261118 08:54:24.005887 855 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal
uuid: "ad2789ff5b54489c84966cff78be4721"
format_stamp: "Formatted at 2026-11-18 08:54:23 on dist-test-slave-m8c8"
I20261118 08:54:24.006083 855 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:24.017926 855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:24.018719 855 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:24.018945 855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:24.019871 855 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261118 08:54:24.021988 855 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261118 08:54:24.022079 855 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:24.022181 855 ts_tablet_manager.cc:662] Registered 0 tablets
I20261118 08:54:24.022250 855 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:24.104028 855 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.129:39287
I20261118 08:54:24.104095 982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.129:39287 every 8 connection(s)
I20261118 08:54:24.105548 855 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/data/info.pb
I20261118 08:54:24.115700 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 855
I20261118 08:54:24.115948 290 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/instance
I20261118 08:54:24.123787 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.72.130:0
--local_ip_for_outbound_sockets=127.0.72.130
--webserver_interface=127.0.72.130
--webserver_port=0
--tserver_master_addrs=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20261118 08:54:24.124344 983 heartbeater.cc:345] Connected to a master server at 127.0.72.188:41259
I20261118 08:54:24.124522 985 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:24.124763 983 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:24.124763 985 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:24.125890 983 heartbeater.cc:508] Master 127.0.72.188:41259 requested a full tablet report, sending...
I20261118 08:54:24.125861 985 heartbeater.cc:508] Master 127.0.72.189:33939 requested a full tablet report, sending...
I20261118 08:54:24.125838 984 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:24.126375 984 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:24.126953 984 heartbeater.cc:508] Master 127.0.72.190:42341 requested a full tablet report, sending...
I20261118 08:54:24.128551 743 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:24.128639 673 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:24.128683 603 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:24.130394 603 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.72.129:51453
I20261118 08:54:24.234964 826 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:24.235057 826 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: loading cluster ID for follower catalog manager: success
I20261118 08:54:24.237641 826 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: acquiring CA information for follower catalog manager: success
I20261118 08:54:24.239377 826 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:24.248214 994 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:24.248431 994 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:24.249696 994 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:24.250236 613 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "857a577b88ba46cc846ecd1a25a722ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78d4026bc230421ea10b8c68a9ef7871" is_pre_election: true
I20261118 08:54:24.250316 753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "857a577b88ba46cc846ecd1a25a722ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" is_pre_election: true
I20261118 08:54:24.250979 657 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:24.251348 994 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:54:24.470762 993 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:24.471072 993 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:24.471210 993 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:24.480082 993 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:24.480270 993 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.72.130
I20261118 08:54:24.490489 993 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.72.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.72.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.72.130
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:24.492208 993 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:24.494048 993 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:24.504308 1000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:24.504444 999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:24.504972 1002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:24.505577 993 server_base.cc:1061] running on GCE node
I20261118 08:54:24.506135 993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:24.507134 993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:24.508333 993 hybrid_clock.cc:648] HybridClock initialized: now 1794992064508259 us; error 68 us; skew 500 ppm
I20261118 08:54:24.508625 993 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:24.510792 993 webserver.cc:577] Webserver started at http://127.0.72.130:45439/ using document root <none> and password file <none>
I20261118 08:54:24.511857 993 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:24.511994 993 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:24.512414 993 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 08:54:24.515533 993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data/instance:
uuid: "213b5f7a64e74b109a3a9387e5975042"
format_stamp: "Formatted at 2026-11-18 08:54:24 on dist-test-slave-m8c8"
I20261118 08:54:24.516551 993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/instance:
uuid: "213b5f7a64e74b109a3a9387e5975042"
format_stamp: "Formatted at 2026-11-18 08:54:24 on dist-test-slave-m8c8"
I20261118 08:54:24.523669 993 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20261118 08:54:24.527768 1008 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:24.529428 993 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20261118 08:54:24.529603 993 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal
uuid: "213b5f7a64e74b109a3a9387e5975042"
format_stamp: "Formatted at 2026-11-18 08:54:24 on dist-test-slave-m8c8"
I20261118 08:54:24.529810 993 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:24.552639 993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:24.553980 993 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:24.554227 993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:24.555228 993 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261118 08:54:24.557058 993 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261118 08:54:24.557161 993 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:24.557284 993 ts_tablet_manager.cc:662] Registered 0 tablets
I20261118 08:54:24.557360 993 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:24.590534 993 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.130:33649
I20261118 08:54:24.590580 1120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.130:33649 every 8 connection(s)
I20261118 08:54:24.592084 993 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/data/info.pb
I20261118 08:54:24.593792 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 993
I20261118 08:54:24.593997 290 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/instance
I20261118 08:54:24.598877 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.72.131:0
--local_ip_for_outbound_sockets=127.0.72.131
--webserver_interface=127.0.72.131
--webserver_port=0
--tserver_master_addrs=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20261118 08:54:24.611102 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:24.611249 1123 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:24.611558 1122 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:24.611567 1123 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:24.612588 1122 heartbeater.cc:508] Master 127.0.72.190:42341 requested a full tablet report, sending...
I20261118 08:54:24.612615 1123 heartbeater.cc:508] Master 127.0.72.189:33939 requested a full tablet report, sending...
I20261118 08:54:24.613386 1121 heartbeater.cc:345] Connected to a master server at 127.0.72.188:41259
I20261118 08:54:24.613510 1121 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:24.614003 1121 heartbeater.cc:508] Master 127.0.72.188:41259 requested a full tablet report, sending...
I20261118 08:54:24.614339 603 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:24.614418 673 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:24.615123 743 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:24.615378 603 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.72.130:37065
I20261118 08:54:24.633255 984 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:24.799680 1129 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:24.799882 1129 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:24.800585 1129 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:24.801195 753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "857a577b88ba46cc846ecd1a25a722ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" is_pre_election: true
I20261118 08:54:24.801245 613 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "857a577b88ba46cc846ecd1a25a722ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78d4026bc230421ea10b8c68a9ef7871" is_pre_election: true
I20261118 08:54:24.801867 658 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:24.802169 1129 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:54:24.964583 1128 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:24.964875 1128 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:24.964996 1128 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:24.973536 1128 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:24.973742 1128 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.72.131
I20261118 08:54:24.984437 1128 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.72.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.72.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.72.131
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:24.986058 1128 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:24.988209 1128 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:24.998422 1135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:24.998409 1134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:24.998462 1137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:24.999557 1128 server_base.cc:1061] running on GCE node
I20261118 08:54:25.000156 1128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:25.001127 1128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:25.002350 1128 hybrid_clock.cc:648] HybridClock initialized: now 1794992065002297 us; error 47 us; skew 500 ppm
I20261118 08:54:25.002650 1128 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:25.004901 1128 webserver.cc:577] Webserver started at http://127.0.72.131:40181/ using document root <none> and password file <none>
I20261118 08:54:25.005620 1128 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:25.005719 1128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:25.006062 1128 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261118 08:54:25.008559 1128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data/instance:
uuid: "3934fbc351ee4a99add35742e90dfa68"
format_stamp: "Formatted at 2026-11-18 08:54:25 on dist-test-slave-m8c8"
I20261118 08:54:25.009317 1128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/instance:
uuid: "3934fbc351ee4a99add35742e90dfa68"
format_stamp: "Formatted at 2026-11-18 08:54:25 on dist-test-slave-m8c8"
I20261118 08:54:25.014825 1128 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20261118 08:54:25.018841 1143 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:25.020363 1128 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261118 08:54:25.020534 1128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal
uuid: "3934fbc351ee4a99add35742e90dfa68"
format_stamp: "Formatted at 2026-11-18 08:54:25 on dist-test-slave-m8c8"
I20261118 08:54:25.020728 1128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:25.038028 1128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:25.038931 1128 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:25.039170 1128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:25.040112 1128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261118 08:54:25.041785 1128 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261118 08:54:25.041877 1128 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:25.041978 1128 ts_tablet_manager.cc:662] Registered 0 tablets
I20261118 08:54:25.042048 1128 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:25.079561 1128 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.131:46447
I20261118 08:54:25.079612 1255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.131:46447 every 8 connection(s)
I20261118 08:54:25.080983 1128 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/data/info.pb
I20261118 08:54:25.089530 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 1128
I20261118 08:54:25.089743 290 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/instance
I20261118 08:54:25.107687 1256 heartbeater.cc:345] Connected to a master server at 127.0.72.188:41259
I20261118 08:54:25.107784 1258 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:25.107931 1257 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:25.108139 1257 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:25.108134 1256 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:25.108134 1258 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:25.109475 1256 heartbeater.cc:508] Master 127.0.72.188:41259 requested a full tablet report, sending...
I20261118 08:54:25.109488 1257 heartbeater.cc:508] Master 127.0.72.190:42341 requested a full tablet report, sending...
I20261118 08:54:25.109499 1258 heartbeater.cc:508] Master 127.0.72.189:33939 requested a full tablet report, sending...
I20261118 08:54:25.111363 743 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:25.111655 673 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:25.111681 603 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:25.112522 603 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.72.131:45591
I20261118 08:54:25.118196 1122 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:25.119500 290 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20261118 08:54:25.152168 602 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_8b08d3e4ed824f1680377a6d5e7518a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.152191 601 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_1ee6cacf31104df284b253928039759b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.154697 600 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_567d34233a4b440a952301f19a6bcb10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.155061 597 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.155674 599 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_41d62c0cf6b54938b8e1770b859e56e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.159219 598 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_294f03c6875c4da0845888273ea80a2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.154533 603 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.159596 596 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_5f1fb0758cf1413eaed1d8b4717aa902"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.160017 595 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_ef26d2c593994e118ff3d01a0f5a9a1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.160173 594 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38576:
name: "default.Table_bc1f66261ac64a87912e19a32ce0ca44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261118 08:54:25.163425 601 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1ee6cacf31104df284b253928039759b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.163425 602 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8b08d3e4ed824f1680377a6d5e7518a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261118 08:54:25.199602 290 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task3eSafg/build/asan/bin/kudu pid 572
I20261118 08:54:25.201236 633 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 08:54:25.353176 1304 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:25.353583 1304 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:25.355222 1304 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:25.355268 753 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "857a577b88ba46cc846ecd1a25a722ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" is_pre_election: true
I20261118 08:54:25.361577 613 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "857a577b88ba46cc846ecd1a25a722ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "78d4026bc230421ea10b8c68a9ef7871" is_pre_election: true
I20261118 08:54:25.362315 658 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:25.362802 290 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task3eSafg/build/asan/bin/kudu with pid 572
I20261118 08:54:25.367575 1304 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:54:25.381270 1264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.72.190:42341)
I20261118 08:54:25.614893 1257 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
W20261118 08:54:25.616374 1257 heartbeater.cc:648] Failed to heartbeat to 127.0.72.190:42341 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:25.621366 1122 heartbeater.cc:648] Failed to heartbeat to 127.0.72.190:42341 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:25.637907 984 heartbeater.cc:648] Failed to heartbeat to 127.0.72.190:42341 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:25.662045 1308 raft_consensus.cc:499] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:25.662232 1308 raft_consensus.cc:521] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:25.662938 1308 leader_election.cc:290] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939)
W20261118 08:54:25.663913 728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:25.664062 681 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
I20261118 08:54:25.664297 681 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c7d9f52503f41e4bafe2821e6fd6af1 in term 1.
I20261118 08:54:25.664844 728 leader_election.cc:304] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 857a577b88ba46cc846ecd1a25a722ca; no voters:
I20261118 08:54:25.665104 1308 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261118 08:54:25.665249 1308 raft_consensus.cc:499] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:25.665371 1308 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 1 FOLLOWER]: Advancing to term 2
W20261118 08:54:25.665561 728 leader_election.cc:336] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:25.685961 1308 raft_consensus.cc:521] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:25.686921 1308 leader_election.cc:290] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 election: Requested vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939)
I20261118 08:54:25.688292 681 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca"
I20261118 08:54:25.688503 681 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 1 FOLLOWER]: Advancing to term 2
W20261118 08:54:25.689944 728 leader_election.cc:336] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:25.782953 681 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c7d9f52503f41e4bafe2821e6fd6af1 in term 2.
I20261118 08:54:25.783748 728 leader_election.cc:304] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 857a577b88ba46cc846ecd1a25a722ca; no voters: 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:25.784811 1308 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 2 FOLLOWER]: Leader election won for term 2
I20261118 08:54:25.785426 1308 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [term 2 LEADER]: Becoming Leader. State: Replica: 6c7d9f52503f41e4bafe2821e6fd6af1, State: Running, Role: LEADER
I20261118 08:54:25.786054 1308 consensus_queue.cc:238] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:25.790704 1311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7d9f52503f41e4bafe2821e6fd6af1. Latest consensus state: current_term: 2 leader_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:25.791229 1311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:25.792609 1315 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 08:54:25.792673 681 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 2 FOLLOWER]: Refusing update from remote peer 6c7d9f52503f41e4bafe2821e6fd6af1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20261118 08:54:25.793606 728 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 -> Peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Couldn't send request to peer 78d4026bc230421ea10b8c68a9ef7871. Status: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:25.793665 1311 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20261118 08:54:25.795461 1304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7d9f52503f41e4bafe2821e6fd6af1. Latest consensus state: current_term: 2 leader_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:25.795629 1304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:25.796083 1315 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 08:54:25.797735 1315 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:25.798377 1315 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 08:54:25.798554 1304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:25.798746 1304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:25.800001 1315 catalog_manager.cc:1555] Loading token signing keys...
I20261118 08:54:25.800213 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:25.800522 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:25.801040 1315 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: Loaded TSK: 0
I20261118 08:54:25.801913 1315 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 08:54:25.802723 1315 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 08:54:25.803512 1315 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 08:54:25.813833 743 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_bc1f66261ac64a87912e19a32ce0ca44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.815349 742 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_ef26d2c593994e118ff3d01a0f5a9a1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.815629 741 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_5f1fb0758cf1413eaed1d8b4717aa902"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.815860 737 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_1ee6cacf31104df284b253928039759b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.816152 736 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_567d34233a4b440a952301f19a6bcb10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.816012 738 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_8b08d3e4ed824f1680377a6d5e7518a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.815757 740 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.816110 735 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_41d62c0cf6b54938b8e1770b859e56e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.816310 739 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:25.817056 734 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_294f03c6875c4da0845888273ea80a2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261118 08:54:25.822751 743 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bc1f66261ac64a87912e19a32ce0ca44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.826678 735 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_41d62c0cf6b54938b8e1770b859e56e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.827874 741 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5f1fb0758cf1413eaed1d8b4717aa902 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.829022 737 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1ee6cacf31104df284b253928039759b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.830036 742 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ef26d2c593994e118ff3d01a0f5a9a1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.831033 739 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c3f2df1b938946c19c1ca00d8fcbd640 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.832034 734 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_294f03c6875c4da0845888273ea80a2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.832958 738 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8b08d3e4ed824f1680377a6d5e7518a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.833933 740 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d2d73a819dc64fe0badbe6d84293bfff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20261118 08:54:25.834869 736 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_567d34233a4b440a952301f19a6bcb10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261118 08:54:26.268656 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--webserver_interface=127.0.72.190
--webserver_port=41231
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 08:54:26.279533 743 catalog_manager.cc:2215] created HMS catalog entry for table [id=4fcf984156584e30aac26283f71adda2]
I20261118 08:54:26.381810 735 catalog_manager.cc:2215] created HMS catalog entry for table [id=a57cd9d759e94822ae1e9ecb7faad9f7]
I20261118 08:54:26.410624 743 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48432:
table { table_name: "default.Table_bc1f66261ac64a87912e19a32ce0ca44" } new_table_name: "default.Table_7fe6a97ebd294ce68601eaf9f252d0ee" modify_external_catalogs: true
I20261118 08:54:26.422962 1056 tablet_service.cc:1500] Processing CreateTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.424873 1056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043a26a7d5004cc883eab2489a3ac5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.427285 918 tablet_service.cc:1500] Processing CreateTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.429122 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043a26a7d5004cc883eab2489a3ac5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.447082 735 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48432:
table { table_name: "default.Table_41d62c0cf6b54938b8e1770b859e56e8" } new_table_name: "default.Table_eefa15fa899c4b1e9162efce0db7e4af" modify_external_catalogs: true
I20261118 08:54:26.450837 1191 tablet_service.cc:1500] Processing CreateTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.454574 1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043a26a7d5004cc883eab2489a3ac5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.455242 1055 tablet_service.cc:1500] Processing CreateTablet for tablet 8b0f9c9d6eef4b66aaab3bcafd810360 (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.455262 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 8b0f9c9d6eef4b66aaab3bcafd810360 (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.455796 1055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0f9c9d6eef4b66aaab3bcafd810360. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.455821 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0f9c9d6eef4b66aaab3bcafd810360. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.462121 1337 tablet_bootstrap.cc:492] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:26.467058 1337 tablet_bootstrap.cc:654] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.472889 917 tablet_service.cc:1500] Processing CreateTablet for tablet 8b0f9c9d6eef4b66aaab3bcafd810360 (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.473467 1336 tablet_bootstrap.cc:492] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:26.473568 917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0f9c9d6eef4b66aaab3bcafd810360. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.478444 1336 tablet_bootstrap.cc:654] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.480154 1336 log.cc:826] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:26.480752 1337 log.cc:826] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:26.483359 1338 tablet_bootstrap.cc:492] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:26.489260 741 catalog_manager.cc:2215] created HMS catalog entry for table [id=c2ea63be7ece4af4bcfb932bd174e957]
I20261118 08:54:26.495781 1338 tablet_bootstrap.cc:654] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.497337 741 master_service.cc:965] ReplaceTablet: received request to replace tablet 043a26a7d5004cc883eab2489a3ac5ba from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:26.497699 1338 log.cc:826] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:26.502333 1337 tablet_bootstrap.cc:492] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:26.502754 1337 ts_tablet_manager.cc:1449] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.000s
I20261118 08:54:26.516451 918 tablet_service.cc:1500] Processing CreateTablet for tablet 274fd2bd8f844da4904afc02a1b15b95 (DEFAULT_TABLE table=default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.516933 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274fd2bd8f844da4904afc02a1b15b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.517217 1055 tablet_service.cc:1500] Processing CreateTablet for tablet 274fd2bd8f844da4904afc02a1b15b95 (DEFAULT_TABLE table=default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.517638 1055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274fd2bd8f844da4904afc02a1b15b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.518153 1337 raft_consensus.cc:360] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.518580 1337 raft_consensus.cc:386] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.518685 1337 raft_consensus.cc:748] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.522723 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 274fd2bd8f844da4904afc02a1b15b95 (DEFAULT_TABLE table=default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.523186 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274fd2bd8f844da4904afc02a1b15b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.528038 741 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: Sending DeleteTablet for 3 replicas of tablet 043a26a7d5004cc883eab2489a3ac5ba
I20261118 08:54:26.529388 1056 tablet_service.cc:1547] Processing DeleteTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba with delete_type TABLET_DATA_DELETED (replaced by tablet 641a0b1632204ce89ed10fa1ad6f8408) from {username='slave'} at 127.0.0.1:52346
I20261118 08:54:26.529937 741 catalog_manager.cc:6936] ReplaceTablet: tablet 043a26a7d5004cc883eab2489a3ac5ba deleted and replaced by tablet 641a0b1632204ce89ed10fa1ad6f8408
I20261118 08:54:26.530058 917 tablet_service.cc:1547] Processing DeleteTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba with delete_type TABLET_DATA_DELETED (replaced by tablet 641a0b1632204ce89ed10fa1ad6f8408) from {username='slave'} at 127.0.0.1:44724
I20261118 08:54:26.530090 1191 tablet_service.cc:1547] Processing DeleteTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba with delete_type TABLET_DATA_DELETED (replaced by tablet 641a0b1632204ce89ed10fa1ad6f8408) from {username='slave'} at 127.0.0.1:59210
I20261118 08:54:26.535815 1337 consensus_queue.cc:261] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.540349 741 master_service.cc:965] ReplaceTablet: received request to replace tablet 043a26a7d5004cc883eab2489a3ac5ba from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:26.542833 918 tablet_service.cc:1500] Processing CreateTablet for tablet 641a0b1632204ce89ed10fa1ad6f8408 (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.542965 741 master_service.cc:965] ReplaceTablet: received request to replace tablet 043a26a7d5004cc883eab2489a3ac5ba from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:26.543412 741 master_service.cc:965] ReplaceTablet: received request to replace tablet 043a26a7d5004cc883eab2489a3ac5ba from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:26.543782 741 master_service.cc:965] ReplaceTablet: received request to replace tablet 043a26a7d5004cc883eab2489a3ac5ba from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:26.544134 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a0b1632204ce89ed10fa1ad6f8408. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.546213 1054 tablet_service.cc:1500] Processing CreateTablet for tablet 641a0b1632204ce89ed10fa1ad6f8408 (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.546729 1054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a0b1632204ce89ed10fa1ad6f8408. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.548483 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 641a0b1632204ce89ed10fa1ad6f8408 (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.548491 741 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48432:
table { table_name: "default.Table_5f1fb0758cf1413eaed1d8b4717aa902" } new_table_name: "default.Table_d9954c97f0054586ab7cf9e5e98b3fa4" modify_external_catalogs: true
I20261118 08:54:26.548903 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a0b1632204ce89ed10fa1ad6f8408. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.552098 1336 tablet_bootstrap.cc:492] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:26.552551 1336 ts_tablet_manager.cc:1449] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.009s
I20261118 08:54:26.553936 1338 tablet_bootstrap.cc:492] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:26.555884 1337 ts_tablet_manager.cc:1480] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.053s user 0.012s sys 0.000s
W20261118 08:54:26.558378 729 catalog_manager.cc:5045] TS 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): delete failed for tablet 043a26a7d5004cc883eab2489a3ac5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 043a26a7d5004cc883eab2489a3ac5ba already in progress: creating tablet
I20261118 08:54:26.563860 1338 ts_tablet_manager.cc:1449] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.081s user 0.000s sys 0.013s
I20261118 08:54:26.556705 1121 heartbeater.cc:500] Master 127.0.72.188:41259 was elected leader, sending a full tablet report...
I20261118 08:54:26.564761 1336 raft_consensus.cc:360] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.565192 1336 raft_consensus.cc:386] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.565330 1336 raft_consensus.cc:748] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.566435 1337 tablet_bootstrap.cc:492] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
W20261118 08:54:26.566852 730 catalog_manager.cc:5045] TS ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): delete failed for tablet 043a26a7d5004cc883eab2489a3ac5ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 043a26a7d5004cc883eab2489a3ac5ba already in progress: creating tablet
I20261118 08:54:26.568606 1336 consensus_queue.cc:261] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.572038 1342 tablet_replica.cc:333] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:26.572589 1342 raft_consensus.cc:2261] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:26.573064 1342 raft_consensus.cc:2290] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:26.574049 1337 tablet_bootstrap.cc:654] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.580785 1342 ts_tablet_manager.cc:1962] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:26.590091 1337 tablet_bootstrap.cc:492] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:26.590378 1337 ts_tablet_manager.cc:1449] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.004s
I20261118 08:54:26.591488 1337 raft_consensus.cc:360] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.591763 1337 raft_consensus.cc:386] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.591907 1337 raft_consensus.cc:748] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.592355 1337 consensus_queue.cc:261] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.593730 1337 ts_tablet_manager.cc:1480] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:26.594102 1337 tablet_bootstrap.cc:492] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:26.595371 1338 raft_consensus.cc:360] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.596056 1337 tablet_bootstrap.cc:654] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
W20261118 08:54:26.597558 1124 tablet.cc:2441] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 08:54:26.603547 1338 raft_consensus.cc:386] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.603704 1338 raft_consensus.cc:748] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.604493 1338 consensus_queue.cc:261] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.608196 1336 ts_tablet_manager.cc:1480] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.055s user 0.020s sys 0.006s
W20261118 08:54:26.611773 986 tablet.cc:2441] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 08:54:26.611979 983 heartbeater.cc:500] Master 127.0.72.188:41259 was elected leader, sending a full tablet report...
I20261118 08:54:26.613361 1337 tablet_bootstrap.cc:492] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:26.613554 1337 ts_tablet_manager.cc:1449] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20261118 08:54:26.614715 1337 raft_consensus.cc:360] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.614925 1337 raft_consensus.cc:386] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.615006 1337 raft_consensus.cc:748] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.615288 1337 consensus_queue.cc:261] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.616365 1338 ts_tablet_manager.cc:1480] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.049s user 0.010s sys 0.000s
I20261118 08:54:26.616532 1337 ts_tablet_manager.cc:1480] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:26.616667 1256 heartbeater.cc:500] Master 127.0.72.188:41259 was elected leader, sending a full tablet report...
I20261118 08:54:26.616865 1337 tablet_bootstrap.cc:492] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:26.619091 1337 tablet_bootstrap.cc:654] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.622673 1338 tablet_bootstrap.cc:492] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:26.624017 1336 tablet_bootstrap.cc:492] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:26.624466 1338 tablet_bootstrap.cc:654] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.626225 1336 tablet_bootstrap.cc:654] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.635164 1336 tablet_bootstrap.cc:492] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:26.635505 1336 ts_tablet_manager.cc:1449] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20261118 08:54:26.635982 1337 tablet_bootstrap.cc:492] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:26.636225 1337 ts_tablet_manager.cc:1449] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20261118 08:54:26.636746 1336 raft_consensus.cc:360] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.637014 1336 raft_consensus.cc:386] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.637019 1337 raft_consensus.cc:360] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.637149 1336 raft_consensus.cc:748] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.637306 1337 raft_consensus.cc:386] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.637449 1337 raft_consensus.cc:748] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.637465 1336 consensus_queue.cc:261] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.637740 1337 consensus_queue.cc:261] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.638027 1336 ts_tablet_manager.cc:1480] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 08:54:26.639055 1337 ts_tablet_manager.cc:1480] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20261118 08:54:26.641839 1338 tablet_bootstrap.cc:492] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:26.642074 1338 ts_tablet_manager.cc:1449] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20261118 08:54:26.643150 1338 raft_consensus.cc:360] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.643435 1338 raft_consensus.cc:386] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.643594 1338 raft_consensus.cc:748] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.643900 1338 consensus_queue.cc:261] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.646380 1338 ts_tablet_manager.cc:1480] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 08:54:26.646910 1338 tablet_bootstrap.cc:492] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:26.648949 1338 tablet_bootstrap.cc:654] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.652192 1336 tablet_bootstrap.cc:492] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:26.654280 1336 tablet_bootstrap.cc:654] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.662907 1336 tablet_bootstrap.cc:492] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:26.663211 1336 ts_tablet_manager.cc:1449] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20261118 08:54:26.663223 1338 tablet_bootstrap.cc:492] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:26.663437 1338 ts_tablet_manager.cc:1449] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20261118 08:54:26.664147 1336 raft_consensus.cc:360] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.664285 1338 raft_consensus.cc:360] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.664433 1336 raft_consensus.cc:386] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.664554 1336 raft_consensus.cc:748] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.664572 1338 raft_consensus.cc:386] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.664661 1338 raft_consensus.cc:748] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.664928 1336 consensus_queue.cc:261] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.665016 1338 consensus_queue.cc:261] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.667377 1338 ts_tablet_manager.cc:1480] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20261118 08:54:26.668778 1338 tablet_bootstrap.cc:492] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:26.670697 1338 tablet_bootstrap.cc:654] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.672175 1336 ts_tablet_manager.cc:1480] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20261118 08:54:26.677323 1336 tablet_bootstrap.cc:492] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:26.679226 1336 tablet_bootstrap.cc:654] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.684708 1338 tablet_bootstrap.cc:492] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:26.684986 1338 ts_tablet_manager.cc:1449] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20261118 08:54:26.685914 1338 raft_consensus.cc:360] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.686242 1338 raft_consensus.cc:386] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.686383 1338 raft_consensus.cc:748] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.686746 1338 consensus_queue.cc:261] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.690821 1338 ts_tablet_manager.cc:1480] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20261118 08:54:26.696619 737 catalog_manager.cc:2215] created HMS catalog entry for table [id=1950821cc4b84c97939b8da6baaab97a]
I20261118 08:54:26.701596 1336 tablet_bootstrap.cc:492] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:26.707162 1336 ts_tablet_manager.cc:1449] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.004s
I20261118 08:54:26.708218 1336 raft_consensus.cc:360] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.708500 1336 raft_consensus.cc:386] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.708647 1336 raft_consensus.cc:748] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.708950 1336 consensus_queue.cc:261] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.711335 1336 ts_tablet_manager.cc:1480] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 08:54:26.718924 918 tablet_service.cc:1500] Processing CreateTablet for tablet 3e292319f0c44e34b49ae6d579df2ecb (DEFAULT_TABLE table=default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.719245 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 3e292319f0c44e34b49ae6d579df2ecb (DEFAULT_TABLE table=default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.719416 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e292319f0c44e34b49ae6d579df2ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.719707 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e292319f0c44e34b49ae6d579df2ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.724362 1056 tablet_service.cc:1500] Processing CreateTablet for tablet 3e292319f0c44e34b49ae6d579df2ecb (DEFAULT_TABLE table=default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:26.727682 917 tablet_service.cc:1547] Processing DeleteTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba with delete_type TABLET_DATA_DELETED (replaced by tablet 641a0b1632204ce89ed10fa1ad6f8408) from {username='slave'} at 127.0.0.1:44724
I20261118 08:54:26.728070 1343 tablet_replica.cc:333] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:26.728155 1191 tablet_service.cc:1547] Processing DeleteTablet for tablet 043a26a7d5004cc883eab2489a3ac5ba with delete_type TABLET_DATA_DELETED (replaced by tablet 641a0b1632204ce89ed10fa1ad6f8408) from {username='slave'} at 127.0.0.1:59210
I20261118 08:54:26.728551 1344 tablet_replica.cc:333] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:26.728605 1343 raft_consensus.cc:2261] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:26.729055 1343 raft_consensus.cc:2290] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:26.729144 1344 raft_consensus.cc:2261] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:26.729202 1338 tablet_bootstrap.cc:492] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:26.729574 1344 raft_consensus.cc:2290] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:26.730993 1338 tablet_bootstrap.cc:654] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.731021 1343 ts_tablet_manager.cc:1962] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:26.733340 1344 ts_tablet_manager.cc:1962] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:26.739281 1336 tablet_bootstrap.cc:492] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:26.741262 1336 tablet_bootstrap.cc:654] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.756130 737 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_0327d29a2f19450b872c54c554d7654e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:26.757633 1336 tablet_bootstrap.cc:492] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:26.757836 1336 ts_tablet_manager.cc:1449] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.002s
I20261118 08:54:26.758687 1336 raft_consensus.cc:360] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.758905 1336 raft_consensus.cc:386] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.758976 1336 raft_consensus.cc:748] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.759316 1336 consensus_queue.cc:261] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.759984 1336 ts_tablet_manager.cc:1480] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20261118 08:54:26.761982 1338 tablet_bootstrap.cc:492] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:26.762231 1338 ts_tablet_manager.cc:1449] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.007s
I20261118 08:54:26.763080 1338 raft_consensus.cc:360] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.763352 1338 raft_consensus.cc:386] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.763525 1338 raft_consensus.cc:748] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.763681 1346 raft_consensus.cc:499] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:26.763936 1338 consensus_queue.cc:261] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.763913 1346 raft_consensus.cc:521] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.765875 1338 ts_tablet_manager.cc:1480] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:26.766124 1346 leader_election.cc:290] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:26.774869 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "274fd2bd8f844da4904afc02a1b15b95" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:26.775404 1076 raft_consensus.cc:2490] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:26.776202 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "274fd2bd8f844da4904afc02a1b15b95" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:26.776614 1211 raft_consensus.cc:2490] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:26.777374 873 leader_election.cc:304] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:26.778245 1346 raft_consensus.cc:2826] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:26.778388 1346 raft_consensus.cc:499] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:26.778475 1346 raft_consensus.cc:3096] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.791992 1361 raft_consensus.cc:499] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:26.792295 1361 raft_consensus.cc:521] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.793228 1361 leader_election.cc:290] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:26.795630 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "3e292319f0c44e34b49ae6d579df2ecb" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:26.795881 1211 raft_consensus.cc:2490] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:26.797363 873 leader_election.cc:304] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
W20261118 08:54:26.797839 873 leader_election.cc:343] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb
I20261118 08:54:26.799099 1342 ts_tablet_manager.cc:1975] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 08:54:26.799279 1342 log.cc:1199] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/043a26a7d5004cc883eab2489a3ac5ba
I20261118 08:54:26.799754 1342 ts_tablet_manager.cc:1996] T 043a26a7d5004cc883eab2489a3ac5ba P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:26.803651 1346 raft_consensus.cc:521] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.804399 1346 leader_election.cc:290] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Requested vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:26.805439 1346 raft_consensus.cc:2826] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:26.805460 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "274fd2bd8f844da4904afc02a1b15b95" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:26.805542 1346 raft_consensus.cc:499] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:26.805644 1346 raft_consensus.cc:3096] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.805673 1076 raft_consensus.cc:3096] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.808547 1056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e292319f0c44e34b49ae6d579df2ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:26.809036 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "274fd2bd8f844da4904afc02a1b15b95" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:26.809253 1211 raft_consensus.cc:3096] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
W20261118 08:54:26.839232 1259 tablet.cc:2441] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 08:54:26.847829 1345 raft_consensus.cc:499] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:26.848053 1345 raft_consensus.cc:521] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.849817 1345 leader_election.cc:290] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:26.858054 1210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "641a0b1632204ce89ed10fa1ad6f8408" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:26.858443 1210 raft_consensus.cc:2490] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:26.858376 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "641a0b1632204ce89ed10fa1ad6f8408" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:26.858783 938 raft_consensus.cc:2490] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:26.859484 1012 leader_election.cc:304] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:26.859932 1345 raft_consensus.cc:2826] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:26.860069 1345 raft_consensus.cc:499] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:26.860174 1345 raft_consensus.cc:3096] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.877820 1344 ts_tablet_manager.cc:1975] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 08:54:26.878036 1344 log.cc:1199] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/043a26a7d5004cc883eab2489a3ac5ba
I20261118 08:54:26.878469 1344 ts_tablet_manager.cc:1996] T 043a26a7d5004cc883eab2489a3ac5ba P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:26.878851 1076 raft_consensus.cc:2490] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:26.879498 1346 raft_consensus.cc:521] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.880347 1346 leader_election.cc:290] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Requested vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:26.880785 1343 ts_tablet_manager.cc:1975] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 08:54:26.880930 1343 log.cc:1199] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/043a26a7d5004cc883eab2489a3ac5ba
I20261118 08:54:26.881358 1343 ts_tablet_manager.cc:1996] T 043a26a7d5004cc883eab2489a3ac5ba P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:26.882301 729 catalog_manager.cc:5065] TS 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): tablet 043a26a7d5004cc883eab2489a3ac5ba (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:26.882730 1337 tablet_bootstrap.cc:492] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:26.879369 1211 raft_consensus.cc:2490] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:26.885074 873 leader_election.cc:304] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:26.885299 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "3e292319f0c44e34b49ae6d579df2ecb" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:26.885406 1346 raft_consensus.cc:2826] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:26.885514 1211 raft_consensus.cc:3096] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.885566 1346 raft_consensus.cc:705] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Becoming Leader. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Running, Role: LEADER
I20261118 08:54:26.885658 1337 tablet_bootstrap.cc:654] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:26.886025 1346 consensus_queue.cc:238] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.888749 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "3e292319f0c44e34b49ae6d579df2ecb" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
W20261118 08:54:26.889840 873 leader_election.cc:343] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Illegal state: must be running to vote when last-logged opid is not known
I20261118 08:54:26.897421 1361 consensus_queue.cc:1049] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:26.898283 1361 consensus_queue.cc:1049] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:26.908226 1337 tablet_bootstrap.cc:492] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:26.908594 1337 ts_tablet_manager.cc:1449] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.007s
I20261118 08:54:26.909608 1337 raft_consensus.cc:360] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.910866 1337 raft_consensus.cc:386] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:26.911443 1337 raft_consensus.cc:748] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:26.911964 1337 consensus_queue.cc:261] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.921526 1337 ts_tablet_manager.cc:1480] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
W20261118 08:54:26.945636 1323 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:26.945983 1323 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:26.946097 1323 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20261118 08:54:26.946841 1349 raft_consensus.cc:499] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:26.947023 1349 raft_consensus.cc:521] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:26.948855 1349 leader_election.cc:290] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:26.953573 730 catalog_manager.cc:5065] TS ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): tablet 043a26a7d5004cc883eab2489a3ac5ba (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:26.958019 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8b0f9c9d6eef4b66aaab3bcafd810360" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:26.958328 938 raft_consensus.cc:2490] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
W20261118 08:54:26.959751 1323 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
I20261118 08:54:26.959733 1345 raft_consensus.cc:521] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
W20261118 08:54:26.960029 1323 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:26.960170 1323 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:26.960276 1323 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:26.960263 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8b0f9c9d6eef4b66aaab3bcafd810360" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:26.960501 1075 raft_consensus.cc:2490] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:26.960566 1345 leader_election.cc:290] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:26.961002 1211 raft_consensus.cc:2490] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:26.962110 1147 leader_election.cc:304] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:26.962657 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "641a0b1632204ce89ed10fa1ad6f8408" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:26.962874 938 raft_consensus.cc:3096] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.963220 1349 raft_consensus.cc:2826] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:26.963327 1349 raft_consensus.cc:499] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:26.963428 1349 raft_consensus.cc:3096] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.966149 873 leader_election.cc:304] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters: 213b5f7a64e74b109a3a9387e5975042
I20261118 08:54:26.966236 729 catalog_manager.cc:5065] TS 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): tablet 043a26a7d5004cc883eab2489a3ac5ba (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:26.966663 1366 raft_consensus.cc:2826] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:26.966799 1366 raft_consensus.cc:705] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Becoming Leader. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Running, Role: LEADER
I20261118 08:54:26.966898 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "641a0b1632204ce89ed10fa1ad6f8408" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:26.967068 1211 raft_consensus.cc:3096] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.967108 1366 consensus_queue.cc:238] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:26.970654 1075 raft_consensus.cc:3096] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:26.971096 1361 consensus_queue.cc:1049] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:26.982343 1323 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.72.190
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:26.984645 1323 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:26.987053 1323 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:26.996796 1383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:26.997169 1384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:26.997388 1386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:26.997841 1323 server_base.cc:1061] running on GCE node
I20261118 08:54:26.998600 1323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:27.000273 1323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:27.001509 1323 hybrid_clock.cc:648] HybridClock initialized: now 1794992067001461 us; error 43 us; skew 500 ppm
I20261118 08:54:27.001868 1323 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:27.004151 1323 webserver.cc:577] Webserver started at http://127.0.72.190:41231/ using document root <none> and password file <none>
I20261118 08:54:27.004866 1323 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:27.004987 1323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:27.012755 1323 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20261118 08:54:27.016367 1392 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:27.017938 1323 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20261118 08:54:27.018126 1323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:27.018846 1323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:27.047648 1323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:27.048566 1323 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:27.048842 1323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:27.055239 938 raft_consensus.cc:2490] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:27.055269 1349 raft_consensus.cc:521] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.056185 1349 leader_election.cc:290] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:27.060043 1211 raft_consensus.cc:2490] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:27.061026 1011 leader_election.cc:304] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters:
I20261118 08:54:27.061384 1345 raft_consensus.cc:2826] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:27.061548 1345 raft_consensus.cc:705] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Becoming Leader. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Running, Role: LEADER
I20261118 08:54:27.061483 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8b0f9c9d6eef4b66aaab3bcafd810360" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:27.061730 1075 raft_consensus.cc:3096] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:27.061991 1345 consensus_queue.cc:238] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.062443 1346 consensus_queue.cc:1049] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:27.065182 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8b0f9c9d6eef4b66aaab3bcafd810360" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:27.065426 938 raft_consensus.cc:3096] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:27.075802 1367 consensus_queue.cc:1049] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:27.076505 1369 consensus_queue.cc:1049] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:27.104101 1323 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.190:42341
I20261118 08:54:27.104101 1447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.190:42341 every 8 connection(s)
I20261118 08:54:27.106247 1323 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
I20261118 08:54:27.113914 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 1323
I20261118 08:54:27.117868 1448 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:27.124105 1448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap starting.
I20261118 08:54:27.136402 1448 log.cc:826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:27.145597 1448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 08:54:27.146621 1448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap complete.
I20261118 08:54:27.151911 1448 raft_consensus.cc:360] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:27.152612 1448 raft_consensus.cc:748] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.153448 1075 raft_consensus.cc:2490] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:27.153451 938 raft_consensus.cc:2490] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:27.153380 1448 consensus_queue.cc:261] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:27.154173 1147 leader_election.cc:304] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:27.154736 1349 raft_consensus.cc:2826] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:27.154886 1349 raft_consensus.cc:705] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 1 LEADER]: Becoming Leader. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Running, Role: LEADER
I20261118 08:54:27.155301 1349 consensus_queue.cc:238] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.162909 1380 consensus_queue.cc:1049] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:27.163647 1380 consensus_queue.cc:1049] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:27.182951 1452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:27.183295 1452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:27.185138 1448 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:27.190968 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:27.191669 1257 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:27.191898 1257 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:27.199574 1257 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:27.207501 984 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:27.222476 1473 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:27.222584 1473 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: loading cluster ID for follower catalog manager: success
I20261118 08:54:27.227195 1473 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: acquiring CA information for follower catalog manager: success
I20261118 08:54:27.229449 1473 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:27.402906 1423 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 1 FOLLOWER]: Advancing to term 2
I20261118 08:54:27.452775 742 catalog_manager.cc:2215] created HMS catalog entry for table [id=19b3ec381a984304badca62634eadd70]
I20261118 08:54:27.475085 1452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7d9f52503f41e4bafe2821e6fd6af1. Latest consensus state: current_term: 2 leader_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:27.475358 1452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:27.476130 1482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:27.476672 1482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:27.492223 1485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:27.492712 1485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:27.502545 742 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48432 (request call id 24) took 1689 ms. Trace:
I20261118 08:54:27.503508 1478 mvcc.cc:204] Tried to move back new op lower bound from 7352287505241210880 to 7352287501465374720. Current Snapshot: MvccSnapshot[applied={T|T < 7352287505241210880 or (T in {7352287505241210880})}]
I20261118 08:54:27.502738 742 rpcz_store.cc:276] 1118 08:54:25.813356 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:25.813435 (+ 79us) service_pool.cc:224] Handling call
1118 08:54:25.830190 (+ 16755us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 08:54:25.830611 (+ 421us) catalog_manager.cc:2188] Created new table and tablet info
1118 08:54:27.452721 (+1622110us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 08:54:27.502250 (+ 49529us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 08:54:27.502367 (+ 117us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 08:54:27.502491 (+ 124us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 08:54:27.455538 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 08:54:27.455703 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
1118 08:54:27.455710 (+ 7us) write_op.cc:435] Acquired schema lock
1118 08:54:27.455713 (+ 3us) tablet.cc:670] Decoding operations
1118 08:54:27.455829 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
1118 08:54:27.455843 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1118 08:54:27.455848 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 08:54:27.455952 (+ 104us) tablet.cc:709] Row locks acquired
1118 08:54:27.455954 (+ 2us) write_op.cc:260] PREPARE: finished
1118 08:54:27.455995 (+ 41us) write_op.cc:270] Start()
1118 08:54:27.456024 (+ 29us) write_op.cc:276] Timestamp: P: 1794992067455868 usec, L: 0
1118 08:54:27.456027 (+ 3us) op_driver.cc:350] REPLICATION: starting
1118 08:54:27.457230 (+ 1203us) log.cc:844] Serialized 432 byte log entry
1118 08:54:27.496144 (+ 38914us) op_driver.cc:466] REPLICATION: finished
1118 08:54:27.501635 (+ 5491us) write_op.cc:301] APPLY: starting
1118 08:54:27.501652 (+ 17us) tablet.cc:1402] starting BulkCheckPresence
1118 08:54:27.501697 (+ 45us) tablet.cc:1405] finished BulkCheckPresence
1118 08:54:27.501769 (+ 72us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 08:54:27.501913 (+ 144us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 08:54:27.501948 (+ 35us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 08:54:27.501956 (+ 8us) write_op.cc:312] APPLY: finished
1118 08:54:27.502049 (+ 93us) log.cc:844] Serialized 28 byte log entry
1118 08:54:27.502121 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:27.502140 (+ 19us) write_op.cc:454] Released schema lock
1118 08:54:27.502146 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":865777,"Hive Metastore.run_cpu_time_us":298,"Hive Metastore.run_wall_time_us":755878,"spinlock_wait_cycles":3968,"child_traces":[["op",{"apply.queue_time_us":5366,"apply.run_cpu_time_us":613,"apply.run_wall_time_us":612,"num_ops":2,"prepare.queue_time_us":1942,"prepare.run_cpu_time_us":907,"prepare.run_wall_time_us":3500,"raft.queue_time_us":45,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":314,"replication_time_us":40047,"thread_start_us":7921,"threads_started":2}]]}
I20261118 08:54:27.523298 1056 tablet_service.cc:1500] Processing CreateTablet for tablet d546b7f20a7f4d3b9e5612080df04be6 (DEFAULT_TABLE table=default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.523836 1056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d546b7f20a7f4d3b9e5612080df04be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.523823 1190 tablet_service.cc:1500] Processing CreateTablet for tablet d546b7f20a7f4d3b9e5612080df04be6 (DEFAULT_TABLE table=default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.524183 918 tablet_service.cc:1500] Processing CreateTablet for tablet d546b7f20a7f4d3b9e5612080df04be6 (DEFAULT_TABLE table=default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.524267 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d546b7f20a7f4d3b9e5612080df04be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.525101 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d546b7f20a7f4d3b9e5612080df04be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.529069 742 catalog_manager.cc:5840] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2789ff5b54489c84966cff78be4721 (127.0.72.129). New cstate: current_term: 1 leader_uuid: "ad2789ff5b54489c84966cff78be4721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } }
I20261118 08:54:27.539335 742 catalog_manager.cc:5840] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2789ff5b54489c84966cff78be4721 (127.0.72.129). New cstate: current_term: 1 leader_uuid: "ad2789ff5b54489c84966cff78be4721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } }
I20261118 08:54:27.540046 742 catalog_manager.cc:5840] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131). New cstate: current_term: 1 leader_uuid: "3934fbc351ee4a99add35742e90dfa68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } health_report { overall_health: HEALTHY } } }
I20261118 08:54:27.540973 742 catalog_manager.cc:5840] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130). New cstate: current_term: 1 leader_uuid: "213b5f7a64e74b109a3a9387e5975042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } }
I20261118 08:54:27.550392 1491 tablet_bootstrap.cc:492] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:27.552649 1491 tablet_bootstrap.cc:654] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.560104 1492 tablet_bootstrap.cc:492] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:27.560722 1493 tablet_bootstrap.cc:492] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:27.565335 742 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_8f6feb81c1374d42bc1e0acb856b75cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:27.565620 1491 tablet_bootstrap.cc:492] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:27.565769 1491 ts_tablet_manager.cc:1449] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20261118 08:54:27.566613 1491 raft_consensus.cc:360] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.566918 1491 raft_consensus.cc:386] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.567050 1491 raft_consensus.cc:748] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.567157 1493 tablet_bootstrap.cc:654] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.567444 1491 consensus_queue.cc:261] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.568045 1491 ts_tablet_manager.cc:1480] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 08:54:27.568998 1492 tablet_bootstrap.cc:654] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.571044 1257 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:27.571219 1257 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:27.575944 1410 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:53371; Asking this server to re-register.
I20261118 08:54:27.576886 1492 tablet_bootstrap.cc:492] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:27.576936 1493 tablet_bootstrap.cc:492] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:27.577065 1257 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:27.577090 1492 ts_tablet_manager.cc:1449] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20261118 08:54:27.577144 1493 ts_tablet_manager.cc:1449] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20261118 08:54:27.577914 1492 raft_consensus.cc:360] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.577925 1493 raft_consensus.cc:360] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.578097 1493 raft_consensus.cc:386] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.578105 1492 raft_consensus.cc:386] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.578205 1492 raft_consensus.cc:748] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.578240 1493 raft_consensus.cc:748] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.578433 1492 consensus_queue.cc:261] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.578974 1492 ts_tablet_manager.cc:1480] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 08:54:27.579208 1410 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:27.580797 1493 consensus_queue.cc:261] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.581216 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:27.582895 1493 ts_tablet_manager.cc:1480] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20261118 08:54:27.584492 984 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:27.585582 1410 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:39561; Asking this server to re-register.
I20261118 08:54:27.586342 1122 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:27.588245 1411 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:46529; Asking this server to re-register.
I20261118 08:54:27.588325 1410 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:27.589033 984 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:27.590528 1410 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:27.664546 739 catalog_manager.cc:2215] created HMS catalog entry for table [id=812c3949fc5544c09b17567b3e81977b]
I20261118 08:54:27.672937 739 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48432 (request call id 27) took 1859 ms. Trace:
I20261118 08:54:27.673100 739 rpcz_store.cc:276] 1118 08:54:25.813721 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:25.813768 (+ 47us) service_pool.cc:224] Handling call
1118 08:54:25.831148 (+ 17380us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 08:54:25.831605 (+ 457us) catalog_manager.cc:2188] Created new table and tablet info
1118 08:54:27.664479 (+1832874us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 08:54:27.672607 (+ 8128us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 08:54:27.672754 (+ 147us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 08:54:27.672885 (+ 131us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 08:54:27.665525 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 08:54:27.665696 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
1118 08:54:27.665703 (+ 7us) write_op.cc:435] Acquired schema lock
1118 08:54:27.665706 (+ 3us) tablet.cc:670] Decoding operations
1118 08:54:27.665810 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
1118 08:54:27.665825 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 08:54:27.665832 (+ 7us) tablet.cc:693] Acquiring locks for 2 operations
1118 08:54:27.665933 (+ 101us) tablet.cc:709] Row locks acquired
1118 08:54:27.665936 (+ 3us) write_op.cc:260] PREPARE: finished
1118 08:54:27.665982 (+ 46us) write_op.cc:270] Start()
1118 08:54:27.666030 (+ 48us) write_op.cc:276] Timestamp: P: 1794992067665939 usec, L: 0
1118 08:54:27.666033 (+ 3us) op_driver.cc:350] REPLICATION: starting
1118 08:54:27.666246 (+ 213us) log.cc:844] Serialized 432 byte log entry
1118 08:54:27.669977 (+ 3731us) op_driver.cc:466] REPLICATION: finished
1118 08:54:27.670075 (+ 98us) write_op.cc:301] APPLY: starting
1118 08:54:27.670091 (+ 16us) tablet.cc:1402] starting BulkCheckPresence
1118 08:54:27.670137 (+ 46us) tablet.cc:1405] finished BulkCheckPresence
1118 08:54:27.670163 (+ 26us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 08:54:27.670389 (+ 226us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 08:54:27.670424 (+ 35us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 08:54:27.670433 (+ 9us) write_op.cc:312] APPLY: finished
1118 08:54:27.670522 (+ 89us) log.cc:844] Serialized 28 byte log entry
1118 08:54:27.670588 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:27.670609 (+ 21us) write_op.cc:454] Released schema lock
1118 08:54:27.670616 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 08:54:27.670697 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:27.670701 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1620705,"Hive Metastore.run_cpu_time_us":340,"Hive Metastore.run_wall_time_us":211953,"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":612,"apply.run_wall_time_us":611,"mutex_wait_us":2525,"num_ops":2,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":879,"prepare.run_wall_time_us":879,"raft.queue_time_us":122,"raft.run_cpu_time_us":729,"raft.run_wall_time_us":727,"replication_time_us":3914,"spinlock_wait_cycles":100992}]]}
I20261118 08:54:27.684930 918 tablet_service.cc:1500] Processing CreateTablet for tablet 8285b9829b7d4250a275063f36e1c93a (DEFAULT_TABLE table=default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.685309 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 8285b9829b7d4250a275063f36e1c93a (DEFAULT_TABLE table=default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.685312 1056 tablet_service.cc:1500] Processing CreateTablet for tablet 8285b9829b7d4250a275063f36e1c93a (DEFAULT_TABLE table=default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.685442 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8285b9829b7d4250a275063f36e1c93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.685729 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8285b9829b7d4250a275063f36e1c93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.685768 1056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8285b9829b7d4250a275063f36e1c93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.690616 1454 raft_consensus.cc:499] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:27.690796 1454 raft_consensus.cc:521] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.692180 1491 tablet_bootstrap.cc:492] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:27.692283 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d546b7f20a7f4d3b9e5612080df04be6" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:27.692485 1075 raft_consensus.cc:2490] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:27.693302 1146 leader_election.cc:304] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters:
I20261118 08:54:27.693877 1454 leader_election.cc:290] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:27.694052 1454 raft_consensus.cc:2826] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:27.694155 1454 raft_consensus.cc:499] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:27.694242 1491 tablet_bootstrap.cc:654] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.694907 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d546b7f20a7f4d3b9e5612080df04be6" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:27.695166 938 raft_consensus.cc:2490] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:27.694258 1454 raft_consensus.cc:3096] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:27.696564 1492 tablet_bootstrap.cc:492] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:27.698309 1492 tablet_bootstrap.cc:654] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.710462 1491 tablet_bootstrap.cc:492] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:27.710673 1491 ts_tablet_manager.cc:1449] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20261118 08:54:27.711649 1491 raft_consensus.cc:360] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.711879 1491 raft_consensus.cc:386] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.711948 1491 raft_consensus.cc:748] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.712221 1491 consensus_queue.cc:261] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.712700 1491 ts_tablet_manager.cc:1480] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 08:54:27.712929 1493 tablet_bootstrap.cc:492] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:27.714751 1493 tablet_bootstrap.cc:654] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.725594 1492 tablet_bootstrap.cc:492] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:27.725852 1492 ts_tablet_manager.cc:1449] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.017s
I20261118 08:54:27.726055 739 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_82d1541f550a4d5482cff56810c97a22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:27.726696 1492 raft_consensus.cc:360] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.726918 1492 raft_consensus.cc:386] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.726989 1492 raft_consensus.cc:748] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.727262 1492 consensus_queue.cc:261] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.732569 1492 ts_tablet_manager.cc:1480] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20261118 08:54:27.734016 1493 tablet_bootstrap.cc:492] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:27.734349 1493 ts_tablet_manager.cc:1449] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20261118 08:54:27.735267 1493 raft_consensus.cc:360] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.735587 1493 raft_consensus.cc:386] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.735723 1493 raft_consensus.cc:748] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.736085 1493 consensus_queue.cc:261] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.739147 1493 ts_tablet_manager.cc:1480] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 08:54:27.762724 1454 raft_consensus.cc:521] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.763480 1454 leader_election.cc:290] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Requested vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:27.764048 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d546b7f20a7f4d3b9e5612080df04be6" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:27.764281 1075 raft_consensus.cc:3096] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:27.764261 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d546b7f20a7f4d3b9e5612080df04be6" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:27.764451 938 raft_consensus.cc:3096] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:27.767246 1075 raft_consensus.cc:2490] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:27.767246 938 raft_consensus.cc:2490] T d546b7f20a7f4d3b9e5612080df04be6 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:27.767839 1147 leader_election.cc:304] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:27.768180 1454 raft_consensus.cc:2826] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:27.768344 1454 raft_consensus.cc:705] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [term 1 LEADER]: Becoming Leader. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Running, Role: LEADER
I20261118 08:54:27.768659 1454 consensus_queue.cc:238] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.774053 1497 consensus_queue.cc:1049] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:27.774451 1497 consensus_queue.cc:1049] T d546b7f20a7f4d3b9e5612080df04be6 P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:27.865644 734 catalog_manager.cc:2215] created HMS catalog entry for table [id=0146c51b5d004bd6b63f5e1c9e1dcf30]
I20261118 08:54:27.872510 734 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48432 (request call id 32) took 2056 ms. Trace:
I20261118 08:54:27.872675 734 rpcz_store.cc:276] 1118 08:54:25.815462 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:25.815528 (+ 66us) service_pool.cc:224] Handling call
1118 08:54:25.832166 (+ 16638us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 08:54:25.832580 (+ 414us) catalog_manager.cc:2188] Created new table and tablet info
1118 08:54:27.865559 (+2032979us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 08:54:27.871669 (+ 6110us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 08:54:27.871797 (+ 128us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 08:54:27.871918 (+ 121us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 08:54:27.866413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 08:54:27.866550 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1118 08:54:27.866556 (+ 6us) write_op.cc:435] Acquired schema lock
1118 08:54:27.866559 (+ 3us) tablet.cc:670] Decoding operations
1118 08:54:27.866657 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
1118 08:54:27.866671 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1118 08:54:27.866677 (+ 6us) tablet.cc:693] Acquiring locks for 2 operations
1118 08:54:27.866805 (+ 128us) tablet.cc:709] Row locks acquired
1118 08:54:27.866807 (+ 2us) write_op.cc:260] PREPARE: finished
1118 08:54:27.866850 (+ 43us) write_op.cc:270] Start()
1118 08:54:27.866888 (+ 38us) write_op.cc:276] Timestamp: P: 1794992067866768 usec, L: 0
1118 08:54:27.866892 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 08:54:27.867096 (+ 204us) log.cc:844] Serialized 432 byte log entry
1118 08:54:27.871007 (+ 3911us) op_driver.cc:466] REPLICATION: finished
1118 08:54:27.871117 (+ 110us) write_op.cc:301] APPLY: starting
1118 08:54:27.871135 (+ 18us) tablet.cc:1402] starting BulkCheckPresence
1118 08:54:27.871182 (+ 47us) tablet.cc:1405] finished BulkCheckPresence
1118 08:54:27.871219 (+ 37us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 08:54:27.871340 (+ 121us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 08:54:27.871404 (+ 64us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 08:54:27.871413 (+ 9us) write_op.cc:312] APPLY: finished
1118 08:54:27.871492 (+ 79us) log.cc:844] Serialized 28 byte log entry
1118 08:54:27.871554 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:27.871575 (+ 21us) write_op.cc:454] Released schema lock
1118 08:54:27.871582 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 08:54:27.871649 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:27.871653 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1831784,"Hive Metastore.run_cpu_time_us":332,"Hive Metastore.run_wall_time_us":200977,"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":522,"num_ops":2,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":816,"prepare.run_wall_time_us":817,"raft.queue_time_us":590,"raft.run_cpu_time_us":659,"raft.run_wall_time_us":657,"replication_time_us":4084}]]}
I20261118 08:54:27.883325 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 39e7203496e0458687be52c3fadd15ea (DEFAULT_TABLE table=default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.883539 918 tablet_service.cc:1500] Processing CreateTablet for tablet 39e7203496e0458687be52c3fadd15ea (DEFAULT_TABLE table=default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.883710 1056 tablet_service.cc:1500] Processing CreateTablet for tablet 39e7203496e0458687be52c3fadd15ea (DEFAULT_TABLE table=default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:27.883848 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e7203496e0458687be52c3fadd15ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.883984 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e7203496e0458687be52c3fadd15ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.884114 1056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39e7203496e0458687be52c3fadd15ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:27.890349 1491 tablet_bootstrap.cc:492] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:27.890872 1493 tablet_bootstrap.cc:492] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:27.892396 1491 tablet_bootstrap.cc:654] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.893070 1493 tablet_bootstrap.cc:654] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.896503 734 catalog_manager.cc:5840] T d546b7f20a7f4d3b9e5612080df04be6 P 213b5f7a64e74b109a3a9387e5975042 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131). New cstate: current_term: 1 leader_uuid: "3934fbc351ee4a99add35742e90dfa68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } }
I20261118 08:54:27.901801 1491 tablet_bootstrap.cc:492] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:27.901988 1491 ts_tablet_manager.cc:1449] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20261118 08:54:27.902786 1491 raft_consensus.cc:360] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.903002 1491 raft_consensus.cc:386] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.903102 1491 raft_consensus.cc:748] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.903354 1491 consensus_queue.cc:261] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.903939 1491 ts_tablet_manager.cc:1480] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 08:54:27.908757 1493 tablet_bootstrap.cc:492] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:27.908947 1493 ts_tablet_manager.cc:1449] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20261118 08:54:27.909128 734 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48432:
table { table_name: "default.Table_1ee6cacf31104df284b253928039759b" } new_table_name: "default.Table_29f622723bbb4a47970de8920bf7d580" modify_external_catalogs: true
I20261118 08:54:27.909822 1493 raft_consensus.cc:360] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.910045 1493 raft_consensus.cc:386] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.910143 1493 raft_consensus.cc:748] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.911229 1493 consensus_queue.cc:261] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.913341 1493 ts_tablet_manager.cc:1480] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20261118 08:54:27.914512 1492 tablet_bootstrap.cc:492] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:27.916465 1492 tablet_bootstrap.cc:654] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:27.919970 1492 tablet_bootstrap.cc:492] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:27.920140 1492 ts_tablet_manager.cc:1449] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20261118 08:54:27.920828 1492 raft_consensus.cc:360] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.920986 1492 raft_consensus.cc:386] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:27.921068 1492 raft_consensus.cc:748] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:27.921312 1492 consensus_queue.cc:261] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.923293 1492 ts_tablet_manager.cc:1480] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:27.943678 1509 raft_consensus.cc:499] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:27.943918 1509 raft_consensus.cc:521] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:27.944746 1509 leader_election.cc:290] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:27.945997 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8285b9829b7d4250a275063f36e1c93a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:27.946024 1210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8285b9829b7d4250a275063f36e1c93a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:27.946249 1210 raft_consensus.cc:2490] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:27.946274 1075 raft_consensus.cc:2490] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:27.947023 873 leader_election.cc:304] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:27.947624 1509 raft_consensus.cc:2826] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:27.947746 1509 raft_consensus.cc:499] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:27.947844 1509 raft_consensus.cc:3096] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.019199 1509 raft_consensus.cc:521] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.020187 1509 leader_election.cc:290] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Requested vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:28.021335 1210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8285b9829b7d4250a275063f36e1c93a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:28.021559 1210 raft_consensus.cc:3096] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.021826 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8285b9829b7d4250a275063f36e1c93a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:28.022025 1075 raft_consensus.cc:3096] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.024044 1075 raft_consensus.cc:2490] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:28.024519 873 leader_election.cc:304] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.024911 1509 raft_consensus.cc:2826] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:28.025055 1509 raft_consensus.cc:705] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Becoming Leader. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Running, Role: LEADER
I20261118 08:54:28.025384 1509 consensus_queue.cc:238] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.030257 1509 consensus_queue.cc:1049] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
08:54:28.079 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_8b08d3e4ed824f1680377a6d5e7518a8 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
W20261118 08:54:28.081851 738 catalog_manager.cc:2211] Already present: failed to create HMS catalog entry for table [id=977eb4910cb14ba3a41f40644a618555]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_8b08d3e4ed824f1680377a6d5e7518a8 already exists)
I20261118 08:54:28.082399 738 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48432 (request call id 28) took 2268 ms. Trace:
I20261118 08:54:28.082540 738 rpcz_store.cc:276] 1118 08:54:25.814151 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:25.814641 (+ 490us) service_pool.cc:224] Handling call
1118 08:54:25.833084 (+ 18443us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 08:54:25.833548 (+ 464us) catalog_manager.cc:2188] Created new table and tablet info
1118 08:54:28.082351 (+2248803us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":2031844,"Hive Metastore.run_cpu_time_us":990,"Hive Metastore.run_wall_time_us":216154}
I20261118 08:54:28.094864 738 catalog_manager.cc:5840] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2789ff5b54489c84966cff78be4721 (127.0.72.129). New cstate: current_term: 1 leader_uuid: "ad2789ff5b54489c84966cff78be4721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } health_report { overall_health: UNKNOWN } } }
I20261118 08:54:28.122166 1210 raft_consensus.cc:2490] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:28.123993 738 master_service.cc:965] ReplaceTablet: received request to replace tablet 043a26a7d5004cc883eab2489a3ac5ba from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:28.124179 1510 consensus_queue.cc:1049] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:28.124475 738 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48432 (request call id 45) took 1380 ms. Trace:
I20261118 08:54:28.124589 738 rpcz_store.cc:276] 1118 08:54:26.744113 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:28.123891 (+1379778us) service_pool.cc:224] Handling call
1118 08:54:28.124436 (+ 545us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 08:54:28.125015 738 master_service.cc:965] ReplaceTablet: received request to replace tablet 641a0b1632204ce89ed10fa1ad6f8408 from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:28.140023 738 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: Sending DeleteTablet for 3 replicas of tablet 641a0b1632204ce89ed10fa1ad6f8408
I20261118 08:54:28.140522 738 catalog_manager.cc:6936] ReplaceTablet: tablet 641a0b1632204ce89ed10fa1ad6f8408 deleted and replaced by tablet 4a6c27574dfb45a1b736aad9c209669e
I20261118 08:54:28.140697 738 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48432 (request call id 46) took 1386 ms. Trace:
I20261118 08:54:28.140825 918 tablet_service.cc:1547] Processing DeleteTablet for tablet 641a0b1632204ce89ed10fa1ad6f8408 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6c27574dfb45a1b736aad9c209669e) from {username='slave'} at 127.0.0.1:44724
I20261118 08:54:28.140998 1056 tablet_service.cc:1547] Processing DeleteTablet for tablet 641a0b1632204ce89ed10fa1ad6f8408 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6c27574dfb45a1b736aad9c209669e) from {username='slave'} at 127.0.0.1:52346
I20261118 08:54:28.140786 738 rpcz_store.cc:276] 1118 08:54:26.754023 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:28.124956 (+1370933us) service_pool.cc:224] Handling call
1118 08:54:28.139697 (+ 14741us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:28.139703 (+ 6us) write_op.cc:454] Released schema lock
1118 08:54:28.140638 (+ 935us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 08:54:28.129079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 08:54:28.129235 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
1118 08:54:28.129242 (+ 7us) write_op.cc:435] Acquired schema lock
1118 08:54:28.129244 (+ 2us) tablet.cc:670] Decoding operations
1118 08:54:28.129373 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
1118 08:54:28.129388 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 08:54:28.129394 (+ 6us) tablet.cc:693] Acquiring locks for 2 operations
1118 08:54:28.129482 (+ 88us) tablet.cc:709] Row locks acquired
1118 08:54:28.129485 (+ 3us) write_op.cc:260] PREPARE: finished
1118 08:54:28.129523 (+ 38us) write_op.cc:270] Start()
1118 08:54:28.129559 (+ 36us) write_op.cc:276] Timestamp: P: 1794992068129481 usec, L: 0
1118 08:54:28.129563 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 08:54:28.129794 (+ 231us) log.cc:844] Serialized 658 byte log entry
1118 08:54:28.137069 (+ 7275us) op_driver.cc:466] REPLICATION: finished
1118 08:54:28.137179 (+ 110us) write_op.cc:301] APPLY: starting
1118 08:54:28.137198 (+ 19us) tablet.cc:1402] starting BulkCheckPresence
1118 08:54:28.137245 (+ 47us) tablet.cc:1405] finished BulkCheckPresence
1118 08:54:28.137291 (+ 46us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 08:54:28.137409 (+ 118us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 08:54:28.137470 (+ 61us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1118 08:54:28.137477 (+ 7us) write_op.cc:312] APPLY: finished
1118 08:54:28.137561 (+ 84us) log.cc:844] Serialized 28 byte log entry
1118 08:54:28.137627 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:28.137649 (+ 22us) write_op.cc:454] Released schema lock
1118 08:54:28.137656 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":535,"apply.run_wall_time_us":535,"num_ops":2,"prepare.queue_time_us":3004,"prepare.run_cpu_time_us":851,"prepare.run_wall_time_us":853,"raft.queue_time_us":519,"raft.run_cpu_time_us":561,"raft.run_wall_time_us":560,"replication_time_us":7473}]]}
I20261118 08:54:28.141782 1190 tablet_service.cc:1547] Processing DeleteTablet for tablet 641a0b1632204ce89ed10fa1ad6f8408 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a6c27574dfb45a1b736aad9c209669e) from {username='slave'} at 127.0.0.1:59210
I20261118 08:54:28.142937 738 master_service.cc:965] ReplaceTablet: received request to replace tablet 274fd2bd8f844da4904afc02a1b15b95 from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:28.152129 917 tablet_service.cc:1500] Processing CreateTablet for tablet 4a6c27574dfb45a1b736aad9c209669e (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.152640 917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6c27574dfb45a1b736aad9c209669e. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.154605 1518 tablet_replica.cc:333] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:28.154810 1518 raft_consensus.cc:2261] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.155038 1518 raft_consensus.cc:2290] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.155725 1518 ts_tablet_manager.cc:1962] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.156180 1520 tablet_replica.cc:333] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:28.156330 1520 raft_consensus.cc:2261] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.156502 1520 raft_consensus.cc:2290] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.156965 1520 ts_tablet_manager.cc:1962] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.159461 1054 tablet_service.cc:1500] Processing CreateTablet for tablet 4a6c27574dfb45a1b736aad9c209669e (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.159759 1519 tablet_replica.cc:333] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:28.159974 1519 raft_consensus.cc:2261] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:28.160058 1054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6c27574dfb45a1b736aad9c209669e. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.160552 1519 raft_consensus.cc:2290] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.161211 1519 ts_tablet_manager.cc:1962] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.163789 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 4a6c27574dfb45a1b736aad9c209669e (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.167006 1492 tablet_bootstrap.cc:492] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:28.167438 738 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 6c7d9f52503f41e4bafe2821e6fd6af1: Sending DeleteTablet for 3 replicas of tablet 274fd2bd8f844da4904afc02a1b15b95
I20261118 08:54:28.168105 738 catalog_manager.cc:6936] ReplaceTablet: tablet 274fd2bd8f844da4904afc02a1b15b95 deleted and replaced by tablet d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:28.168291 738 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48432 (request call id 47) took 1412 ms. Trace:
I20261118 08:54:28.168735 1492 tablet_bootstrap.cc:654] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.170406 1056 tablet_service.cc:1547] Processing DeleteTablet for tablet 274fd2bd8f844da4904afc02a1b15b95 with delete_type TABLET_DATA_DELETED (replaced by tablet d92e535f2bf44815b6c003b56dd25b79) from {username='slave'} at 127.0.0.1:52346
I20261118 08:54:28.170516 1191 tablet_service.cc:1547] Processing DeleteTablet for tablet 274fd2bd8f844da4904afc02a1b15b95 with delete_type TABLET_DATA_DELETED (replaced by tablet d92e535f2bf44815b6c003b56dd25b79) from {username='slave'} at 127.0.0.1:59210
I20261118 08:54:28.168445 738 rpcz_store.cc:276] 1118 08:54:26.755978 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:28.142849 (+1386871us) service_pool.cc:224] Handling call
1118 08:54:28.168256 (+ 25407us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 08:54:28.145009 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 08:54:28.145128 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
1118 08:54:28.145134 (+ 6us) write_op.cc:435] Acquired schema lock
1118 08:54:28.145136 (+ 2us) tablet.cc:670] Decoding operations
1118 08:54:28.145269 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
1118 08:54:28.145302 (+ 33us) write_op.cc:641] Partition lock acquired for write op
1118 08:54:28.145307 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 08:54:28.145388 (+ 81us) tablet.cc:709] Row locks acquired
1118 08:54:28.145391 (+ 3us) write_op.cc:260] PREPARE: finished
1118 08:54:28.145423 (+ 32us) write_op.cc:270] Start()
1118 08:54:28.145452 (+ 29us) write_op.cc:276] Timestamp: P: 1794992068145379 usec, L: 0
1118 08:54:28.145455 (+ 3us) op_driver.cc:350] REPLICATION: starting
1118 08:54:28.145643 (+ 188us) log.cc:844] Serialized 658 byte log entry
1118 08:54:28.149315 (+ 3672us) op_driver.cc:466] REPLICATION: finished
1118 08:54:28.149979 (+ 664us) write_op.cc:301] APPLY: starting
1118 08:54:28.149989 (+ 10us) tablet.cc:1402] starting BulkCheckPresence
1118 08:54:28.150030 (+ 41us) tablet.cc:1405] finished BulkCheckPresence
1118 08:54:28.150051 (+ 21us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 08:54:28.150134 (+ 83us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 08:54:28.150186 (+ 52us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1118 08:54:28.150192 (+ 6us) write_op.cc:312] APPLY: finished
1118 08:54:28.150257 (+ 65us) log.cc:844] Serialized 28 byte log entry
1118 08:54:28.150309 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:28.150325 (+ 16us) write_op.cc:454] Released schema lock
1118 08:54:28.150330 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":626,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":412,"num_ops":2,"prepare.queue_time_us":1108,"prepare.run_cpu_time_us":723,"prepare.run_wall_time_us":722,"raft.queue_time_us":431,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":3835,"thread_start_us":7997,"threads_started":1}]]}
I20261118 08:54:28.171923 918 tablet_service.cc:1547] Processing DeleteTablet for tablet 274fd2bd8f844da4904afc02a1b15b95 with delete_type TABLET_DATA_DELETED (replaced by tablet d92e535f2bf44815b6c003b56dd25b79) from {username='slave'} at 127.0.0.1:44724
I20261118 08:54:28.174630 1493 tablet_bootstrap.cc:492] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:28.175695 738 master_service.cc:965] ReplaceTablet: received request to replace tablet 043a26a7d5004cc883eab2489a3ac5ba from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:28.176476 1493 tablet_bootstrap.cc:654] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.177129 738 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48432 (request call id 48) took 1421 ms. Trace:
I20261118 08:54:28.177230 1518 ts_tablet_manager.cc:1975] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.177234 738 rpcz_store.cc:276] 1118 08:54:26.756093 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:28.175615 (+1419522us) service_pool.cc:224] Handling call
1118 08:54:28.176892 (+ 1277us) spinlock_profiling.cc:245] Waited 921 us on lock 0x616000041b54. stack: 00007fa1036fb3a6 00007fa1036fb160 00007fa1026ec34d 00007fa119cbf704 00007fa119cbf6cd 00007fa112c1182d 00007fa112c11923 00007fa10f77b221 00007fa11767050d 00007fa11780c6fb 00007fa10e114e67 00007fa10e114b95 00007fa10a8438de 00007fa10a8427be 00007fa10a8489f8 00007fa10a84935f
1118 08:54:28.177071 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118528}
I20261118 08:54:28.177429 1518 log.cc:1199] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/641a0b1632204ce89ed10fa1ad6f8408
I20261118 08:54:28.177527 738 master_service.cc:965] ReplaceTablet: received request to replace tablet 641a0b1632204ce89ed10fa1ad6f8408 from {username='slave'} at 127.0.0.1:48432
I20261118 08:54:28.178197 738 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48432 (request call id 49) took 1421 ms. Trace:
I20261118 08:54:28.178303 738 rpcz_store.cc:276] 1118 08:54:26.756228 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:28.177449 (+1421221us) service_pool.cc:224] Handling call
1118 08:54:28.178164 (+ 715us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20261118 08:54:28.178840 1518 ts_tablet_manager.cc:1996] T 641a0b1632204ce89ed10fa1ad6f8408 P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:28.179606 1492 tablet_bootstrap.cc:492] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:28.179767 1492 ts_tablet_manager.cc:1449] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20261118 08:54:28.179831 1518 tablet_replica.cc:333] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:28.180027 1518 raft_consensus.cc:2261] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:28.180552 1518 raft_consensus.cc:2290] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.181121 1518 ts_tablet_manager.cc:1962] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.180882 1492 raft_consensus.cc:360] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.181602 1493 tablet_bootstrap.cc:492] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:28.181710 1492 raft_consensus.cc:386] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.181756 1493 ts_tablet_manager.cc:1449] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20261118 08:54:28.181795 1492 raft_consensus.cc:748] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.182077 1492 consensus_queue.cc:261] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.183019 1520 ts_tablet_manager.cc:1975] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.183153 1520 log.cc:1199] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/641a0b1632204ce89ed10fa1ad6f8408
I20261118 08:54:28.183568 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6c27574dfb45a1b736aad9c209669e. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.183632 1520 ts_tablet_manager.cc:1996] T 641a0b1632204ce89ed10fa1ad6f8408 P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:28.184224 1191 tablet_service.cc:1500] Processing CreateTablet for tablet d92e535f2bf44815b6c003b56dd25b79 (DEFAULT_TABLE table=default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.184623 1520 tablet_replica.cc:333] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:28.184652 1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92e535f2bf44815b6c003b56dd25b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.185087 1519 ts_tablet_manager.cc:1975] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.185187 1519 log.cc:1199] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/641a0b1632204ce89ed10fa1ad6f8408
I20261118 08:54:28.185590 1519 ts_tablet_manager.cc:1996] T 641a0b1632204ce89ed10fa1ad6f8408 P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:28.186549 1519 tablet_replica.cc:333] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:28.186731 1519 raft_consensus.cc:2261] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.186797 729 catalog_manager.cc:5065] TS 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): tablet 641a0b1632204ce89ed10fa1ad6f8408 (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:28.186899 1519 raft_consensus.cc:2290] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.187322 729 catalog_manager.cc:5065] TS 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): tablet 641a0b1632204ce89ed10fa1ad6f8408 (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:28.187465 1519 ts_tablet_manager.cc:1962] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.187819 1054 tablet_service.cc:1500] Processing CreateTablet for tablet d92e535f2bf44815b6c003b56dd25b79 (DEFAULT_TABLE table=default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.188201 1054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92e535f2bf44815b6c003b56dd25b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.190033 1491 tablet_bootstrap.cc:492] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:28.184827 1520 raft_consensus.cc:2261] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.191581 1520 raft_consensus.cc:2290] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.191668 1492 ts_tablet_manager.cc:1480] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20261118 08:54:28.191924 1491 tablet_bootstrap.cc:654] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.192224 1520 ts_tablet_manager.cc:1962] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.195540 1491 tablet_bootstrap.cc:492] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:28.195708 1491 ts_tablet_manager.cc:1449] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20261118 08:54:28.192262 738 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "default.Table_903cefd8f2a74c89be4bf41b84d26ba4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:28.196487 1491 raft_consensus.cc:360] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.196684 1491 raft_consensus.cc:386] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.196754 1491 raft_consensus.cc:748] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.197453 1491 consensus_queue.cc:261] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.198319 1520 ts_tablet_manager.cc:1975] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.198464 1520 log.cc:1199] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/274fd2bd8f844da4904afc02a1b15b95
I20261118 08:54:28.198895 1520 ts_tablet_manager.cc:1996] T 274fd2bd8f844da4904afc02a1b15b95 P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:28.199752 1491 ts_tablet_manager.cc:1480] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20261118 08:54:28.200069 1491 tablet_bootstrap.cc:492] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:28.200425 1492 tablet_bootstrap.cc:492] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:28.202378 1492 tablet_bootstrap.cc:654] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.202404 1491 tablet_bootstrap.cc:654] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.204087 730 catalog_manager.cc:5065] TS ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): tablet 641a0b1632204ce89ed10fa1ad6f8408 (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:28.205286 1493 raft_consensus.cc:360] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.205479 1493 raft_consensus.cc:386] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.205538 1493 raft_consensus.cc:748] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.205804 1493 consensus_queue.cc:261] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.206096 740 catalog_manager.cc:2215] created HMS catalog entry for table [id=fa8c2c5f6be14649a1bd9abaca0167c8]
I20261118 08:54:28.206244 1519 ts_tablet_manager.cc:1975] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.206388 1519 log.cc:1199] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/274fd2bd8f844da4904afc02a1b15b95
I20261118 08:54:28.207355 729 catalog_manager.cc:5065] TS 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): tablet 274fd2bd8f844da4904afc02a1b15b95 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]) successfully deleted
I20261118 08:54:28.207969 1519 ts_tablet_manager.cc:1996] T 274fd2bd8f844da4904afc02a1b15b95 P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:28.209700 729 catalog_manager.cc:5065] TS 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): tablet 274fd2bd8f844da4904afc02a1b15b95 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]) successfully deleted
I20261118 08:54:28.209659 917 tablet_service.cc:1500] Processing CreateTablet for tablet d92e535f2bf44815b6c003b56dd25b79 (DEFAULT_TABLE table=default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.210067 917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92e535f2bf44815b6c003b56dd25b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.210307 1493 ts_tablet_manager.cc:1480] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20261118 08:54:28.217330 1493 tablet_bootstrap.cc:492] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:28.218408 1518 ts_tablet_manager.cc:1975] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.219112 1518 log.cc:1199] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/274fd2bd8f844da4904afc02a1b15b95
I20261118 08:54:28.219676 1518 ts_tablet_manager.cc:1996] T 274fd2bd8f844da4904afc02a1b15b95 P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:28.219807 1493 tablet_bootstrap.cc:654] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.220896 730 catalog_manager.cc:5065] TS ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): tablet 274fd2bd8f844da4904afc02a1b15b95 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]) successfully deleted
I20261118 08:54:28.221029 740 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48432 (request call id 26) took 2407 ms. Trace:
I20261118 08:54:28.221148 740 rpcz_store.cc:276] 1118 08:54:25.813605 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:25.813693 (+ 88us) service_pool.cc:224] Handling call
1118 08:54:25.834062 (+ 20369us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 08:54:25.834455 (+ 393us) catalog_manager.cc:2188] Created new table and tablet info
1118 08:54:28.206043 (+2371588us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 08:54:28.220729 (+ 14686us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 08:54:28.220872 (+ 143us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 08:54:28.220987 (+ 115us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 08:54:28.206851 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 08:54:28.206978 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
1118 08:54:28.206985 (+ 7us) write_op.cc:435] Acquired schema lock
1118 08:54:28.206987 (+ 2us) tablet.cc:670] Decoding operations
1118 08:54:28.207083 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1118 08:54:28.207097 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1118 08:54:28.207101 (+ 4us) tablet.cc:693] Acquiring locks for 2 operations
1118 08:54:28.207187 (+ 86us) tablet.cc:709] Row locks acquired
1118 08:54:28.207189 (+ 2us) write_op.cc:260] PREPARE: finished
1118 08:54:28.207240 (+ 51us) write_op.cc:270] Start()
1118 08:54:28.207277 (+ 37us) write_op.cc:276] Timestamp: P: 1794992068207180 usec, L: 0
1118 08:54:28.207281 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 08:54:28.207526 (+ 245us) log.cc:844] Serialized 432 byte log entry
1118 08:54:28.216598 (+ 9072us) op_driver.cc:466] REPLICATION: finished
1118 08:54:28.216744 (+ 146us) write_op.cc:301] APPLY: starting
1118 08:54:28.216765 (+ 21us) tablet.cc:1402] starting BulkCheckPresence
1118 08:54:28.216820 (+ 55us) tablet.cc:1405] finished BulkCheckPresence
1118 08:54:28.216875 (+ 55us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 08:54:28.217001 (+ 126us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 08:54:28.217040 (+ 39us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 08:54:28.217048 (+ 8us) write_op.cc:312] APPLY: finished
1118 08:54:28.217144 (+ 96us) log.cc:844] Serialized 28 byte log entry
1118 08:54:28.217217 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:28.217242 (+ 25us) write_op.cc:454] Released schema lock
1118 08:54:28.217250 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1118 08:54:28.217328 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:28.217332 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":2247138,"Hive Metastore.run_cpu_time_us":288,"Hive Metastore.run_wall_time_us":124254,"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":574,"apply.run_wall_time_us":572,"num_ops":2,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":799,"prepare.run_wall_time_us":799,"raft.queue_time_us":3815,"raft.run_cpu_time_us":553,"raft.run_wall_time_us":553,"replication_time_us":9279}]]}
I20261118 08:54:28.227355 1492 tablet_bootstrap.cc:492] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:28.227538 1492 ts_tablet_manager.cc:1449] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20261118 08:54:28.227716 1491 tablet_bootstrap.cc:492] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:28.227869 1491 ts_tablet_manager.cc:1449] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.004s
I20261118 08:54:28.228314 1492 raft_consensus.cc:360] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.228502 1492 raft_consensus.cc:386] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.228582 1492 raft_consensus.cc:748] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.228645 1491 raft_consensus.cc:360] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.228943 1491 raft_consensus.cc:386] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.229043 1491 raft_consensus.cc:748] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.229279 1491 consensus_queue.cc:261] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.231776 740 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48432:
table { table_name: "default.Table_ef26d2c593994e118ff3d01a0f5a9a1d" } new_table_name: "default.Table_0d932fab32a1414582d1f8370b6e2b23" modify_external_catalogs: true
I20261118 08:54:28.232607 1491 ts_tablet_manager.cc:1480] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20261118 08:54:28.233337 1190 tablet_service.cc:1500] Processing CreateTablet for tablet a810b27a15594c708bfed8f4a5129912 (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.233852 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a810b27a15594c708bfed8f4a5129912. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.233889 917 tablet_service.cc:1500] Processing CreateTablet for tablet a810b27a15594c708bfed8f4a5129912 (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.228813 1492 consensus_queue.cc:261] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.233160 1054 tablet_service.cc:1500] Processing CreateTablet for tablet a810b27a15594c708bfed8f4a5129912 (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.234294 917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a810b27a15594c708bfed8f4a5129912. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.234439 1054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a810b27a15594c708bfed8f4a5129912. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.235597 1454 raft_consensus.cc:499] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:28.236354 1454 raft_consensus.cc:521] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.237030 1454 leader_election.cc:290] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:28.237749 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "39e7203496e0458687be52c3fadd15ea" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:28.237957 1075 raft_consensus.cc:2490] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:28.238700 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "39e7203496e0458687be52c3fadd15ea" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:28.238946 938 raft_consensus.cc:2490] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:28.239482 1147 leader_election.cc:304] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.240729 1454 raft_consensus.cc:2826] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:28.240823 1454 raft_consensus.cc:499] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:28.240921 1454 raft_consensus.cc:3096] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.243543 1492 ts_tablet_manager.cc:1480] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20261118 08:54:28.245167 1493 tablet_bootstrap.cc:492] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:28.245335 1493 ts_tablet_manager.cc:1449] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.002s
I20261118 08:54:28.245249 1454 raft_consensus.cc:521] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.246146 1493 raft_consensus.cc:360] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.246351 1493 raft_consensus.cc:386] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.246366 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "39e7203496e0458687be52c3fadd15ea" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:28.246592 938 raft_consensus.cc:3096] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.246668 1492 tablet_bootstrap.cc:492] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:28.248759 1454 leader_election.cc:290] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:28.249338 938 raft_consensus.cc:2490] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:28.249993 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "39e7203496e0458687be52c3fadd15ea" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:28.250187 1075 raft_consensus.cc:3096] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.250285 1147 leader_election.cc:304] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.250734 1454 raft_consensus.cc:2826] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:28.250932 1454 raft_consensus.cc:705] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [term 1 LEADER]: Becoming Leader. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Running, Role: LEADER
I20261118 08:54:28.251276 1454 consensus_queue.cc:238] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.253623 1075 raft_consensus.cc:2490] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:28.254199 1491 tablet_bootstrap.cc:492] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:28.256068 1491 tablet_bootstrap.cc:654] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.258052 1491 tablet_bootstrap.cc:492] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:28.258232 1491 ts_tablet_manager.cc:1449] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20261118 08:54:28.259035 1491 raft_consensus.cc:360] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.259244 1491 raft_consensus.cc:386] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.259326 1491 raft_consensus.cc:748] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.246452 1493 raft_consensus.cc:748] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.259644 1491 consensus_queue.cc:261] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.259927 1493 consensus_queue.cc:261] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.260182 1491 ts_tablet_manager.cc:1480] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261118 08:54:28.263559 1493 ts_tablet_manager.cc:1480] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20261118 08:54:28.263891 1493 tablet_bootstrap.cc:492] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:28.265749 1493 tablet_bootstrap.cc:654] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.267858 1492 tablet_bootstrap.cc:654] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.271222 1502 consensus_queue.cc:1049] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:28.274747 1493 tablet_bootstrap.cc:492] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:28.274937 1493 ts_tablet_manager.cc:1449] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20261118 08:54:28.275234 1502 consensus_queue.cc:1049] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:28.275804 1493 raft_consensus.cc:360] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.277097 1493 raft_consensus.cc:386] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.277186 1493 raft_consensus.cc:748] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.277467 1493 consensus_queue.cc:261] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.278142 1493 ts_tablet_manager.cc:1480] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:28.279613 736 catalog_manager.cc:2215] created HMS catalog entry for table [id=39ea4f28a4744d1290fe1e9246789b0a]
I20261118 08:54:28.285796 736 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48432 (request call id 30) took 2471 ms. Trace:
I20261118 08:54:28.285941 736 rpcz_store.cc:276] 1118 08:54:25.814632 (+ 0us) service_pool.cc:167] Inserting onto call queue
1118 08:54:25.814896 (+ 264us) service_pool.cc:224] Handling call
1118 08:54:25.834997 (+ 20101us) catalog_manager.cc:2142] Acquired catalog manager lock
1118 08:54:25.835465 (+ 468us) catalog_manager.cc:2188] Created new table and tablet info
1118 08:54:28.279564 (+2444099us) catalog_manager.cc:2214] Created new table in HMS catalog
1118 08:54:28.285509 (+ 5945us) catalog_manager.cc:2248] Wrote table and tablets to system table
1118 08:54:28.285626 (+ 117us) catalog_manager.cc:2282] Inserted table and tablets into CatalogManager maps
1118 08:54:28.285744 (+ 118us) inbound_call.cc:177] Queueing success response
Related trace 'op':
1118 08:54:28.280414 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1118 08:54:28.280549 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
1118 08:54:28.280555 (+ 6us) write_op.cc:435] Acquired schema lock
1118 08:54:28.280557 (+ 2us) tablet.cc:670] Decoding operations
1118 08:54:28.280650 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
1118 08:54:28.280665 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1118 08:54:28.280670 (+ 5us) tablet.cc:693] Acquiring locks for 2 operations
1118 08:54:28.280755 (+ 85us) tablet.cc:709] Row locks acquired
1118 08:54:28.280758 (+ 3us) write_op.cc:260] PREPARE: finished
1118 08:54:28.280796 (+ 38us) write_op.cc:270] Start()
1118 08:54:28.280834 (+ 38us) write_op.cc:276] Timestamp: P: 1794992068280723 usec, L: 0
1118 08:54:28.280838 (+ 4us) op_driver.cc:350] REPLICATION: starting
1118 08:54:28.281009 (+ 171us) log.cc:844] Serialized 432 byte log entry
1118 08:54:28.284812 (+ 3803us) op_driver.cc:466] REPLICATION: finished
1118 08:54:28.284944 (+ 132us) write_op.cc:301] APPLY: starting
1118 08:54:28.284963 (+ 19us) tablet.cc:1402] starting BulkCheckPresence
1118 08:54:28.285037 (+ 74us) tablet.cc:1405] finished BulkCheckPresence
1118 08:54:28.285066 (+ 29us) tablet.cc:1407] starting ApplyRowOperation cycle
1118 08:54:28.285185 (+ 119us) tablet.cc:1418] finished ApplyRowOperation cycle
1118 08:54:28.285222 (+ 37us) tablet_metrics.cc:626] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1118 08:54:28.285231 (+ 9us) write_op.cc:312] APPLY: finished
1118 08:54:28.285317 (+ 86us) log.cc:844] Serialized 28 byte log entry
1118 08:54:28.285390 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:28.285414 (+ 24us) write_op.cc:454] Released schema lock
1118 08:54:28.285421 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1118 08:54:28.285501 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
1118 08:54:28.285505 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":2370456,"Hive Metastore.run_cpu_time_us":277,"Hive Metastore.run_wall_time_us":73189,"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":547,"apply.run_wall_time_us":547,"num_ops":2,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":738,"raft.queue_time_us":469,"raft.run_cpu_time_us":464,"raft.run_wall_time_us":463,"replication_time_us":3942}]]}
I20261118 08:54:28.293460 736 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48432:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
I20261118 08:54:28.293871 736 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48432:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
F20261118 08:54:28.299798 823 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_bc1f66261ac64a87912e19a32ce0ca44, message={"server":"","servicePrincipal":"","db":"default","table":"table_bc1f66261ac64a87912e19a32ce0ca44","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_bc1f66261ac64a87912e19a32ce0ca44\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_bc1f66261ac64a87912e19a32ce0ca44\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_bc1f66261ac64a87912e19a32ce0ca44\",\"kudu.table_id\":\"4fcf984156584e30aac26283f71adda2\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_8b08d3e4ed824f1680377a6d5e7518a8, message={"server":"","servicePrincipal":"","db":"default","table":"table_8b08d3e4ed824f1680377a6d5e7518a8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_8b08d3e4ed824f1680377a6d5e7518a8\",\"kudu.table_id\":\"b42eb739941947f8b7d79fddd923eb52\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20261118 08:54:28.305480 917 tablet_service.cc:1500] Processing CreateTablet for tablet 543333f1b0df4517a424aa536902aa29 (DEFAULT_TABLE table=default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.305960 917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543333f1b0df4517a424aa536902aa29. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.306023 1190 tablet_service.cc:1500] Processing CreateTablet for tablet 543333f1b0df4517a424aa536902aa29 (DEFAULT_TABLE table=default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.306418 1190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543333f1b0df4517a424aa536902aa29. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.308720 1054 tablet_service.cc:1500] Processing CreateTablet for tablet 543333f1b0df4517a424aa536902aa29 (DEFAULT_TABLE table=default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.309131 1054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543333f1b0df4517a424aa536902aa29. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.314401 1491 tablet_bootstrap.cc:492] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
@ 0x7fa1022d9d5d google::LogMessage::Fail() at ??:0
I20261118 08:54:28.317199 1491 tablet_bootstrap.cc:654] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.319558 1345 raft_consensus.cc:499] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:28.319738 1345 raft_consensus.cc:521] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.320428 1345 leader_election.cc:290] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:28.321115 937 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4a6c27574dfb45a1b736aad9c209669e" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:28.321344 937 raft_consensus.cc:2490] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:28.321985 1012 leader_election.cc:304] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
@ 0x7fa1022ddb23 google::LogMessage::SendToLog() at ??:0
I20261118 08:54:28.325440 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4a6c27574dfb45a1b736aad9c209669e" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:28.325652 1211 raft_consensus.cc:2490] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:28.326416 1345 raft_consensus.cc:2826] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:28.326537 1345 raft_consensus.cc:499] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:28.326642 1345 raft_consensus.cc:3096] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.329778 1493 tablet_bootstrap.cc:492] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:28.329692 1345 raft_consensus.cc:521] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.330845 937 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4a6c27574dfb45a1b736aad9c209669e" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:28.331046 937 raft_consensus.cc:3096] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.331665 1493 tablet_bootstrap.cc:654] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
@ 0x7fa1022d975c google::LogMessage::Flush() at ??:0
@ 0x7fa1022d99a1 google::LogMessage::~LogMessage() at ??:0
I20261118 08:54:28.334388 937 raft_consensus.cc:2490] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:28.344208 1012 leader_election.cc:304] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.348232 1493 tablet_bootstrap.cc:492] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:28.348404 1493 ts_tablet_manager.cc:1449] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20261118 08:54:28.349267 1493 raft_consensus.cc:360] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.349460 1493 raft_consensus.cc:386] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.349541 1493 raft_consensus.cc:748] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.349860 1493 consensus_queue.cc:261] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.350554 1493 ts_tablet_manager.cc:1480] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
@ 0x7fa1177599a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 08:54:28.353995 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4a6c27574dfb45a1b736aad9c209669e" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:28.354213 1211 raft_consensus.cc:3096] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.357362 1211 raft_consensus.cc:2490] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:28.357738 1345 leader_election.cc:290] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
@ 0x7fa117758251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 08:54:28.358060 1345 raft_consensus.cc:2826] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:28.358201 1345 raft_consensus.cc:705] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Becoming Leader. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Running, Role: LEADER
I20261118 08:54:28.358489 1345 consensus_queue.cc:238] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
@ 0x7fa11775fecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fa11775fd49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
@ 0x7fa103722dc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa106dfd6db start_thread at ??:0
@ 0x7fa0ff86d61f clone at ??:0
I20261118 08:54:28.382212 1345 consensus_queue.cc:1049] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 08:54:28.394656 1012 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.72.129:39287, user_credentials={real_user=slave}} blocked reactor thread for 50484.6us
I20261118 08:54:28.396574 1539 consensus_queue.cc:1049] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261118 08:54:28.416657 1491 tablet_bootstrap.cc:492] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:28.416646 1492 tablet_bootstrap.cc:492] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:28.416805 1491 ts_tablet_manager.cc:1449] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.103s user 0.016s sys 0.029s
I20261118 08:54:28.416821 1492 ts_tablet_manager.cc:1449] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.170s user 0.040s sys 0.074s
I20261118 08:54:28.417572 1491 raft_consensus.cc:360] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.417557 1492 raft_consensus.cc:360] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.417738 1491 raft_consensus.cc:386] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.417750 1492 raft_consensus.cc:386] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.417814 1492 raft_consensus.cc:748] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.417830 1491 raft_consensus.cc:748] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.418040 1492 consensus_queue.cc:261] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.418040 1491 consensus_queue.cc:261] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.418545 1492 ts_tablet_manager.cc:1480] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 08:54:28.418882 1492 tablet_bootstrap.cc:492] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:28.419857 1491 ts_tablet_manager.cc:1480] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20261118 08:54:28.422991 1492 tablet_bootstrap.cc:654] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.432242 1492 tablet_bootstrap.cc:492] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:28.432538 1492 ts_tablet_manager.cc:1449] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20261118 08:54:28.433347 1492 raft_consensus.cc:360] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.433538 1492 raft_consensus.cc:386] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.433610 1492 raft_consensus.cc:748] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.433828 1492 consensus_queue.cc:261] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.434410 1492 ts_tablet_manager.cc:1480] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 08:54:28.527207 1539 raft_consensus.cc:499] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:28.527460 1539 raft_consensus.cc:521] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.528165 1539 leader_election.cc:290] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:28.528745 937 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "543333f1b0df4517a424aa536902aa29" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:28.528935 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "543333f1b0df4517a424aa536902aa29" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:28.529016 937 raft_consensus.cc:2490] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:28.529121 1211 raft_consensus.cc:2490] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:28.529520 1012 leader_election.cc:304] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.529829 1539 raft_consensus.cc:2826] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:28.529918 1539 raft_consensus.cc:499] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:28.530020 1539 raft_consensus.cc:3096] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.531994 1539 raft_consensus.cc:521] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.532712 1539 leader_election.cc:290] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:28.533030 937 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "543333f1b0df4517a424aa536902aa29" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:28.533245 937 raft_consensus.cc:3096] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.533352 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "543333f1b0df4517a424aa536902aa29" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:28.533519 1211 raft_consensus.cc:3096] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.536197 937 raft_consensus.cc:2490] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:28.536348 1211 raft_consensus.cc:2490] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:28.536707 1012 leader_election.cc:304] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.537056 1539 raft_consensus.cc:2826] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:28.537232 1539 raft_consensus.cc:705] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Becoming Leader. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Running, Role: LEADER
I20261118 08:54:28.537504 1539 consensus_queue.cc:238] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.541482 1345 consensus_queue.cc:1049] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:28.541908 1345 consensus_queue.cc:1049] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:28.571550 1510 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:28.571822 1510 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.572649 1510 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:28.573218 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d92e535f2bf44815b6c003b56dd25b79" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:28.573246 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d92e535f2bf44815b6c003b56dd25b79" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:28.573457 1211 raft_consensus.cc:2490] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:28.573524 1076 raft_consensus.cc:2490] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:28.573940 873 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.574236 1510 raft_consensus.cc:2826] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:28.574329 1510 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:28.574433 1510 raft_consensus.cc:3096] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.576426 1510 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.577327 1510 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Requested vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:28.577734 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d92e535f2bf44815b6c003b56dd25b79" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:28.577939 1076 raft_consensus.cc:3096] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.577891 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d92e535f2bf44815b6c003b56dd25b79" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:28.578084 1211 raft_consensus.cc:3096] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:28.580838 1211 raft_consensus.cc:2490] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:28.580852 1076 raft_consensus.cc:2490] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:28.581255 873 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:28.581540 1510 raft_consensus.cc:2826] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:28.582011 1510 raft_consensus.cc:705] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Becoming Leader. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Running, Role: LEADER
I20261118 08:54:28.582310 1510 consensus_queue.cc:238] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:28.586050 1510 consensus_queue.cc:1049] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:28.586565 1510 consensus_queue.cc:1049] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 08:54:28.618249 1263 connection.cc:573] client connection to 127.0.72.188:41259 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:28.619001 1263 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.72.188:41259)
W20261118 08:54:28.622732 871 connection.cc:573] client connection to 127.0.72.188:41259 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:28.625501 1263 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.72.188:41259)
W20261118 08:54:28.626159 1263 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.72.188:41259)
I20261118 08:54:28.618587 1323 master_runner.cc:187] Error getting master registration for 127.0.72.188:41259: Network error: recv got EOF from 127.0.72.188:41259 (error 108),
I20261118 08:54:28.626886 1323 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:54:28.626922 1263 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.72.188:41259)
W20261118 08:54:28.636518 1567 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:28.623648 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.188:41259 (local address 127.0.72.129:43057)
1118 08:54:28.636280 (+ 12632us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:28.636466 (+ 186us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":12485,"threads_started":1}
W20261118 08:54:28.640509 1566 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:28.622461 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.188:41259 (local address 127.0.72.131:49593)
1118 08:54:28.640246 (+ 17785us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:28.640438 (+ 192us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":17652,"thread_start_us":17527,"threads_started":1}
W20261118 08:54:28.642477 1565 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:28.622080 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.188:41259 (local address 127.0.72.130:60761)
1118 08:54:28.642246 (+ 20166us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:28.642427 (+ 181us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":20036,"thread_start_us":19969,"threads_started":1}
I20261118 08:54:28.801988 1569 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6c7d9f52503f41e4bafe2821e6fd6af1)
I20261118 08:54:28.802239 1569 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:28.803051 1570 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6c7d9f52503f41e4bafe2821e6fd6af1)
I20261118 08:54:28.803205 1570 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:28.803932 1570 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:28.804880 1569 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:28.805241 683 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
I20261118 08:54:28.805501 683 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 2.
W20261118 08:54:28.805843 657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 4 similar messages]
I20261118 08:54:28.806042 1394 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters:
I20261118 08:54:28.806595 1569 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261118 08:54:28.806725 1569 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6c7d9f52503f41e4bafe2821e6fd6af1)
I20261118 08:54:28.806834 1569 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 2 FOLLOWER]: Advancing to term 3
W20261118 08:54:28.807274 1393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:28.808490 657 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:28.809656 1393 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:28.810768 1569 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:28.811672 1569 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 3 election: Requested vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:28.812171 683 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca"
I20261118 08:54:28.812387 683 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 2 FOLLOWER]: Advancing to term 3
W20261118 08:54:28.812678 1393 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:28.813727 1423 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "857a577b88ba46cc846ecd1a25a722ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "78d4026bc230421ea10b8c68a9ef7871" is_pre_election: true
I20261118 08:54:28.814146 1423 raft_consensus.cc:2415] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 857a577b88ba46cc846ecd1a25a722ca in current term 3: Already voted for candidate 78d4026bc230421ea10b8c68a9ef7871 in this term.
I20261118 08:54:28.814743 658 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:28.815070 683 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 3.
I20261118 08:54:28.815229 1570 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20261118 08:54:28.815498 1394 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1
I20261118 08:54:28.815780 1569 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Leader election won for term 3
I20261118 08:54:28.816136 1569 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 LEADER]: Becoming Leader. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Running, Role: LEADER
I20261118 08:54:28.816545 1569 consensus_queue.cc:238] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:28.817925 1576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 3 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:28.818229 1576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:28.819013 1579 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 08:54:28.819947 683 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Refusing update from remote peer 78d4026bc230421ea10b8c68a9ef7871: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
W20261118 08:54:28.820395 1393 consensus_peers.cc:597] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 -> Peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Couldn't send request to peer 6c7d9f52503f41e4bafe2821e6fd6af1. Status: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:28.820622 1577 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20261118 08:54:28.822113 1570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 3 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:28.822363 1570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:28.822317 1579 catalog_manager.cc:694] Loaded metadata for table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]
I20261118 08:54:28.822914 1579 catalog_manager.cc:694] Loaded metadata for table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]
I20261118 08:54:28.823493 1579 catalog_manager.cc:694] Loaded metadata for table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]
I20261118 08:54:28.823851 1569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:28.824137 1569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:28.824148 1579 catalog_manager.cc:694] Loaded metadata for table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]
I20261118 08:54:28.824548 1579 catalog_manager.cc:694] Loaded metadata for table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]
I20261118 08:54:28.824970 1579 catalog_manager.cc:694] Loaded metadata for table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]
I20261118 08:54:28.824920 1570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:28.825093 1570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:28.825384 1579 catalog_manager.cc:694] Loaded metadata for table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]
I20261118 08:54:28.825739 1579 catalog_manager.cc:694] Loaded metadata for table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]
I20261118 08:54:28.826082 1579 catalog_manager.cc:694] Loaded metadata for table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]
I20261118 08:54:28.828271 1579 tablet_loader.cc:96] loaded metadata for tablet 39e7203496e0458687be52c3fadd15ea (table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30])
I20261118 08:54:28.828600 1579 tablet_loader.cc:96] loaded metadata for tablet 3e292319f0c44e34b49ae6d579df2ecb (table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a])
I20261118 08:54:28.828903 1579 tablet_loader.cc:96] loaded metadata for tablet 4a6c27574dfb45a1b736aad9c209669e (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2])
I20261118 08:54:28.829214 1579 tablet_loader.cc:96] loaded metadata for tablet 543333f1b0df4517a424aa536902aa29 (table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a])
I20261118 08:54:28.830054 1579 tablet_loader.cc:96] loaded metadata for tablet 8285b9829b7d4250a275063f36e1c93a (table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b])
I20261118 08:54:28.830389 1579 tablet_loader.cc:96] loaded metadata for tablet 8b0f9c9d6eef4b66aaab3bcafd810360 (table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7])
I20261118 08:54:28.830726 1579 tablet_loader.cc:96] loaded metadata for tablet a810b27a15594c708bfed8f4a5129912 (table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8])
I20261118 08:54:28.831005 1579 tablet_loader.cc:96] loaded metadata for tablet d546b7f20a7f4d3b9e5612080df04be6 (table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70])
I20261118 08:54:28.831290 1579 tablet_loader.cc:96] loaded metadata for tablet d92e535f2bf44815b6c003b56dd25b79 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957])
I20261118 08:54:28.831568 1579 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 08:54:28.832562 1579 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:28.832638 1579 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 08:54:28.833827 1579 catalog_manager.cc:1555] Loading token signing keys...
I20261118 08:54:28.834650 1579 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Loaded TSK: 0
I20261118 08:54:28.835618 1579 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 08:54:28.836447 1579 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 08:54:28.837217 1579 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 08:54:28.852720 1411 master_service.cc:965] ReplaceTablet: received request to replace tablet a810b27a15594c708bfed8f4a5129912 from {username='slave'} at 127.0.0.1:54800
I20261118 08:54:28.852623 1410 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54800:
name: "default.Table_52bba9536ae942a59235d8087d61576b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:28.853251 1405 master_service.cc:965] ReplaceTablet: received request to replace tablet a810b27a15594c708bfed8f4a5129912 from {username='slave'} at 127.0.0.1:54800
I20261118 08:54:28.853641 1406 master_service.cc:965] ReplaceTablet: received request to replace tablet 4a6c27574dfb45a1b736aad9c209669e from {username='slave'} at 127.0.0.1:54800
I20261118 08:54:28.855685 1404 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_567d34233a4b440a952301f19a6bcb10" } modify_external_catalogs: true
I20261118 08:54:28.857137 1408 master_service.cc:965] ReplaceTablet: received request to replace tablet 8285b9829b7d4250a275063f36e1c93a from {username='slave'} at 127.0.0.1:54800
I20261118 08:54:28.856966 1407 master_service.cc:965] ReplaceTablet: received request to replace tablet 8b0f9c9d6eef4b66aaab3bcafd810360 from {username='slave'} at 127.0.0.1:54800
I20261118 08:54:28.857741 1409 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54800:
name: "default.Table_bfd124a61a904e60bc99bee8716f64cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:28.857852 1404 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_567d34233a4b440a952301f19a6bcb10" } modify_external_catalogs: true
I20261118 08:54:28.858095 1403 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff" } modify_external_catalogs: true
I20261118 08:54:28.858073 1400 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54800:
name: "default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:28.859889 1403 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff" } modify_external_catalogs: true
F20261118 08:54:28.863770 1471 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_bc1f66261ac64a87912e19a32ce0ca44, message={"server":"","servicePrincipal":"","db":"default","table":"table_bc1f66261ac64a87912e19a32ce0ca44","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_bc1f66261ac64a87912e19a32ce0ca44\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_bc1f66261ac64a87912e19a32ce0ca44\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_bc1f66261ac64a87912e19a32ce0ca44\",\"kudu.table_id\":\"4fcf984156584e30aac26283f71adda2\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_8b08d3e4ed824f1680377a6d5e7518a8, message={"server":"","servicePrincipal":"","db":"default","table":"table_8b08d3e4ed824f1680377a6d5e7518a8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_8b08d3e4ed824f1680377a6d5e7518a8\",\"kudu.table_id\":\"b42eb739941947f8b7d79fddd923eb52\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f60ec331d5d google::LogMessage::Fail() at ??:0
@ 0x7f60ec335b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f60ec33175c google::LogMessage::Flush() at ??:0
@ 0x7f60ec3319a1 google::LogMessage::~LogMessage() at ??:0
I20261118 08:54:28.878122 1406 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 4a6c27574dfb45a1b736aad9c209669e
I20261118 08:54:28.879335 1411 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet a810b27a15594c708bfed8f4a5129912
I20261118 08:54:28.881014 1406 catalog_manager.cc:6936] ReplaceTablet: tablet 4a6c27574dfb45a1b736aad9c209669e deleted and replaced by tablet 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:54:28.881546 1411 catalog_manager.cc:6936] ReplaceTablet: tablet a810b27a15594c708bfed8f4a5129912 deleted and replaced by tablet c02766f2f8d94f479e0ce2dc03567a7d
I20261118 08:54:28.901758 1411 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_294f03c6875c4da0845888273ea80a2a" } modify_external_catalogs: true
I20261118 08:54:28.902024 1411 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_294f03c6875c4da0845888273ea80a2a" } modify_external_catalogs: true
I20261118 08:54:28.902236 918 tablet_service.cc:1547] Processing DeleteTablet for tablet a810b27a15594c708bfed8f4a5129912 with delete_type TABLET_DATA_DELETED (replaced by tablet c02766f2f8d94f479e0ce2dc03567a7d) from {username='slave'} at 127.0.0.1:44738
I20261118 08:54:28.902384 917 tablet_service.cc:1547] Processing DeleteTablet for tablet 4a6c27574dfb45a1b736aad9c209669e with delete_type TABLET_DATA_DELETED (replaced by tablet 8d92c02e29724e80820a7114d8bbdf7a) from {username='slave'} at 127.0.0.1:44738
I20261118 08:54:28.902592 1406 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54800:
name: "default.Table_738b3d8aface40768cac1e7b01d47046"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:28.902884 1405 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54800:
name: "default.Table_3655f04cfdf84b26839c6e76aac6ebbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:28.904846 1407 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 8b0f9c9d6eef4b66aaab3bcafd810360
I20261118 08:54:28.905481 1407 catalog_manager.cc:6936] ReplaceTablet: tablet 8b0f9c9d6eef4b66aaab3bcafd810360 deleted and replaced by tablet fc173583a9584741b28f2c0203504a3a
I20261118 08:54:28.905892 1407 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54800:
name: "default.Table_b5863ec6b7e144f58594f23a0634bf0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:28.906720 917 tablet_service.cc:1547] Processing DeleteTablet for tablet 8b0f9c9d6eef4b66aaab3bcafd810360 with delete_type TABLET_DATA_DELETED (replaced by tablet fc173583a9584741b28f2c0203504a3a) from {username='slave'} at 127.0.0.1:44738
I20261118 08:54:28.907140 1408 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 8285b9829b7d4250a275063f36e1c93a
I20261118 08:54:28.908082 1408 catalog_manager.cc:6936] ReplaceTablet: tablet 8285b9829b7d4250a275063f36e1c93a deleted and replaced by tablet d7f4643e2be049d7b2e3249a40c6b2ac
I20261118 08:54:28.908615 1408 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
I20261118 08:54:28.908906 1408 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54800:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
I20261118 08:54:28.910432 917 tablet_service.cc:1547] Processing DeleteTablet for tablet 8285b9829b7d4250a275063f36e1c93a with delete_type TABLET_DATA_DELETED (replaced by tablet d7f4643e2be049d7b2e3249a40c6b2ac) from {username='slave'} at 127.0.0.1:44738
@ 0x7f61017b19a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 08:54:28.914885 1191 tablet_service.cc:1547] Processing DeleteTablet for tablet a810b27a15594c708bfed8f4a5129912 with delete_type TABLET_DATA_DELETED (replaced by tablet c02766f2f8d94f479e0ce2dc03567a7d) from {username='slave'} at 127.0.0.1:59224
I20261118 08:54:28.914834 1190 tablet_service.cc:1547] Processing DeleteTablet for tablet 4a6c27574dfb45a1b736aad9c209669e with delete_type TABLET_DATA_DELETED (replaced by tablet 8d92c02e29724e80820a7114d8bbdf7a) from {username='slave'} at 127.0.0.1:59224
I20261118 08:54:28.915279 1188 tablet_service.cc:1547] Processing DeleteTablet for tablet 8285b9829b7d4250a275063f36e1c93a with delete_type TABLET_DATA_DELETED (replaced by tablet d7f4643e2be049d7b2e3249a40c6b2ac) from {username='slave'} at 127.0.0.1:59224
I20261118 08:54:28.915498 1189 tablet_service.cc:1547] Processing DeleteTablet for tablet 8b0f9c9d6eef4b66aaab3bcafd810360 with delete_type TABLET_DATA_DELETED (replaced by tablet fc173583a9584741b28f2c0203504a3a) from {username='slave'} at 127.0.0.1:59224
I20261118 08:54:28.919181 1191 tablet_service.cc:1500] Processing CreateTablet for tablet c02766f2f8d94f479e0ce2dc03567a7d (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.919646 1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c02766f2f8d94f479e0ce2dc03567a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.920046 1189 tablet_service.cc:1500] Processing CreateTablet for tablet 8d92c02e29724e80820a7114d8bbdf7a (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.920574 1189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d92c02e29724e80820a7114d8bbdf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.921535 917 tablet_service.cc:1500] Processing CreateTablet for tablet c02766f2f8d94f479e0ce2dc03567a7d (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.921973 917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c02766f2f8d94f479e0ce2dc03567a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.923552 1054 tablet_service.cc:1547] Processing DeleteTablet for tablet 4a6c27574dfb45a1b736aad9c209669e with delete_type TABLET_DATA_DELETED (replaced by tablet 8d92c02e29724e80820a7114d8bbdf7a) from {username='slave'} at 127.0.0.1:52358
I20261118 08:54:28.924041 1595 tablet_replica.cc:333] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:28.924393 1595 raft_consensus.cc:2261] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.924611 1056 tablet_service.cc:1547] Processing DeleteTablet for tablet a810b27a15594c708bfed8f4a5129912 with delete_type TABLET_DATA_DELETED (replaced by tablet c02766f2f8d94f479e0ce2dc03567a7d) from {username='slave'} at 127.0.0.1:52358
I20261118 08:54:28.924877 1055 tablet_service.cc:1547] Processing DeleteTablet for tablet 8b0f9c9d6eef4b66aaab3bcafd810360 with delete_type TABLET_DATA_DELETED (replaced by tablet fc173583a9584741b28f2c0203504a3a) from {username='slave'} at 127.0.0.1:52358
I20261118 08:54:28.924711 1595 raft_consensus.cc:2290] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.925153 916 tablet_service.cc:1500] Processing CreateTablet for tablet 8d92c02e29724e80820a7114d8bbdf7a (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.925642 916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d92c02e29724e80820a7114d8bbdf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.925664 1595 ts_tablet_manager.cc:1962] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.926753 1055 tablet_service.cc:1547] Processing DeleteTablet for tablet 8285b9829b7d4250a275063f36e1c93a with delete_type TABLET_DATA_DELETED (replaced by tablet d7f4643e2be049d7b2e3249a40c6b2ac) from {username='slave'} at 127.0.0.1:52358
I20261118 08:54:28.927448 1055 tablet_service.cc:1500] Processing CreateTablet for tablet 8d92c02e29724e80820a7114d8bbdf7a (DEFAULT_TABLE table=default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.927893 1055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d92c02e29724e80820a7114d8bbdf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.931851 1056 tablet_service.cc:1500] Processing CreateTablet for tablet c02766f2f8d94f479e0ce2dc03567a7d (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.932330 1056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c02766f2f8d94f479e0ce2dc03567a7d. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f61017b0251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 08:54:28.935575 1492 tablet_bootstrap.cc:492] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:28.937563 1492 tablet_bootstrap.cc:654] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.937731 1599 tablet_bootstrap.cc:492] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:28.939713 1599 tablet_bootstrap.cc:654] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
@ 0x7f61017b7ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 08:54:28.941167 1595 ts_tablet_manager.cc:1975] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 08:54:28.941335 1595 log.cc:1199] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/a810b27a15594c708bfed8f4a5129912
I20261118 08:54:28.941763 1595 ts_tablet_manager.cc:1996] T a810b27a15594c708bfed8f4a5129912 P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:28.942346 1599 tablet_bootstrap.cc:492] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:28.942489 1599 ts_tablet_manager.cc:1449] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20261118 08:54:28.943113 1396 catalog_manager.cc:5065] TS ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): tablet a810b27a15594c708bfed8f4a5129912 (table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]) successfully deleted
I20261118 08:54:28.943297 1599 raft_consensus.cc:360] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:28.943533 1599 raft_consensus.cc:386] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.943598 1595 tablet_replica.cc:333] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:28.943629 1599 raft_consensus.cc:748] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.943852 1595 raft_consensus.cc:2261] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.944039 1595 raft_consensus.cc:2290] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.943859 1599 consensus_queue.cc:261] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:28.944686 1599 ts_tablet_manager.cc:1480] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261118 08:54:28.945024 1599 tablet_bootstrap.cc:492] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:28.945751 1598 tablet_replica.cc:333] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:28.945940 1598 raft_consensus.cc:2261] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:28.946322 1598 raft_consensus.cc:2290] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.947116 1598 ts_tablet_manager.cc:1962] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.948472 1599 tablet_bootstrap.cc:654] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.948590 1595 ts_tablet_manager.cc:1962] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.954685 1598 ts_tablet_manager.cc:1975] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.954819 1598 log.cc:1199] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/4a6c27574dfb45a1b736aad9c209669e
I20261118 08:54:28.955248 1598 ts_tablet_manager.cc:1996] T 4a6c27574dfb45a1b736aad9c209669e P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:28.955268 1595 ts_tablet_manager.cc:1975] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.955456 1595 log.cc:1199] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/4a6c27574dfb45a1b736aad9c209669e
I20261118 08:54:28.955902 1595 ts_tablet_manager.cc:1996] T 4a6c27574dfb45a1b736aad9c209669e P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:28.956182 1598 tablet_replica.cc:333] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:28.956259 1395 catalog_manager.cc:5065] TS 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): tablet 4a6c27574dfb45a1b736aad9c209669e (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:28.956377 1598 raft_consensus.cc:2261] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.956533 1598 raft_consensus.cc:2290] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.957016 1598 ts_tablet_manager.cc:1962] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.957012 1396 catalog_manager.cc:5065] TS ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): tablet 4a6c27574dfb45a1b736aad9c209669e (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]) successfully deleted
I20261118 08:54:28.957006 1189 tablet_service.cc:1500] Processing CreateTablet for tablet fc173583a9584741b28f2c0203504a3a (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.957404 1595 tablet_replica.cc:333] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:28.957459 1189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc173583a9584741b28f2c0203504a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.957588 1595 raft_consensus.cc:2261] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.957661 1054 tablet_service.cc:1500] Processing CreateTablet for tablet fc173583a9584741b28f2c0203504a3a (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.958022 1054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc173583a9584741b28f2c0203504a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.958320 918 tablet_service.cc:1500] Processing CreateTablet for tablet fc173583a9584741b28f2c0203504a3a (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.958752 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc173583a9584741b28f2c0203504a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.961407 1191 tablet_service.cc:1500] Processing CreateTablet for tablet d7f4643e2be049d7b2e3249a40c6b2ac (DEFAULT_TABLE table=default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.961794 1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f4643e2be049d7b2e3249a40c6b2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.963734 1055 tablet_service.cc:1500] Processing CreateTablet for tablet d7f4643e2be049d7b2e3249a40c6b2ac (DEFAULT_TABLE table=default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.964154 1055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f4643e2be049d7b2e3249a40c6b2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.967142 1599 tablet_bootstrap.cc:492] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:28.967324 1599 ts_tablet_manager.cc:1449] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20261118 08:54:28.967584 1597 tablet_replica.cc:333] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:28.967784 1597 raft_consensus.cc:2261] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.968007 1597 raft_consensus.cc:2290] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.968329 1599 raft_consensus.cc:360] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:28.968506 1597 ts_tablet_manager.cc:1962] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.968546 1599 raft_consensus.cc:386] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.968627 1599 raft_consensus.cc:748] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.968957 1599 consensus_queue.cc:261] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:28.971697 1599 ts_tablet_manager.cc:1480] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
@ 0x7f61017b7d49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 08:54:28.973032 1599 tablet_bootstrap.cc:492] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:28.973680 916 tablet_service.cc:1500] Processing CreateTablet for tablet d7f4643e2be049d7b2e3249a40c6b2ac (DEFAULT_TABLE table=default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:28.974110 916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f4643e2be049d7b2e3249a40c6b2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:28.957769 1595 raft_consensus.cc:2290] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.975972 1599 tablet_bootstrap.cc:654] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.976070 1595 ts_tablet_manager.cc:1962] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f60ed77adc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60f0e556db start_thread at ??:0
I20261118 08:54:28.979650 1604 tablet_bootstrap.cc:492] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:28.981792 1492 tablet_bootstrap.cc:492] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:28.981971 1492 ts_tablet_manager.cc:1449] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.003s
I20261118 08:54:28.982764 1492 raft_consensus.cc:360] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:28.983376 1492 raft_consensus.cc:386] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.983482 1492 raft_consensus.cc:748] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.983771 1492 consensus_queue.cc:261] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:28.984412 1599 tablet_bootstrap.cc:492] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:28.984561 1599 ts_tablet_manager.cc:1449] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20261118 08:54:28.985205 1598 ts_tablet_manager.cc:1975] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 08:54:28.985333 1598 log.cc:1199] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/a810b27a15594c708bfed8f4a5129912
I20261118 08:54:28.985391 1599 raft_consensus.cc:360] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:28.985625 1599 raft_consensus.cc:386] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:28.985680 1598 ts_tablet_manager.cc:1996] T a810b27a15594c708bfed8f4a5129912 P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:28.985721 1599 raft_consensus.cc:748] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:28.986002 1599 consensus_queue.cc:261] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:28.986531 1598 tablet_replica.cc:333] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:28.986706 1598 raft_consensus.cc:2261] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:28.986865 1598 raft_consensus.cc:2290] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f60e98c561f clone at ??:0
I20261118 08:54:28.988775 1595 ts_tablet_manager.cc:1975] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.988931 1595 log.cc:1199] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/8b0f9c9d6eef4b66aaab3bcafd810360
I20261118 08:54:28.989209 1599 ts_tablet_manager.cc:1480] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20261118 08:54:28.989393 1595 ts_tablet_manager.cc:1996] T 8b0f9c9d6eef4b66aaab3bcafd810360 P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:28.990067 1492 ts_tablet_manager.cc:1480] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20261118 08:54:28.990407 1595 tablet_replica.cc:333] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:28.990641 1595 raft_consensus.cc:2261] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:28.990988 1595 raft_consensus.cc:2290] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:28.991914 1595 ts_tablet_manager.cc:1962] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.992282 1599 tablet_bootstrap.cc:492] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:28.992913 1598 ts_tablet_manager.cc:1962] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:28.994145 1599 tablet_bootstrap.cc:654] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.994863 1492 tablet_bootstrap.cc:492] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:28.996616 1492 tablet_bootstrap.cc:654] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:28.998629 1595 ts_tablet_manager.cc:1975] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.998812 1595 log.cc:1199] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/8285b9829b7d4250a275063f36e1c93a
I20261118 08:54:28.999225 1595 ts_tablet_manager.cc:1996] T 8285b9829b7d4250a275063f36e1c93a P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:28.999255 1598 ts_tablet_manager.cc:1975] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:28.999365 1598 log.cc:1199] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/8b0f9c9d6eef4b66aaab3bcafd810360
I20261118 08:54:28.999778 1598 ts_tablet_manager.cc:1996] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:29.000658 1598 tablet_replica.cc:333] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:29.000841 1598 raft_consensus.cc:2261] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:29.000993 1598 raft_consensus.cc:2290] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:29.001811 1598 ts_tablet_manager.cc:1962] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:29.002457 1597 ts_tablet_manager.cc:1975] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20261118 08:54:29.002573 1597 log.cc:1199] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/a810b27a15594c708bfed8f4a5129912
I20261118 08:54:29.002689 1604 tablet_bootstrap.cc:654] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:29.003016 1597 ts_tablet_manager.cc:1996] T a810b27a15594c708bfed8f4a5129912 P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:29.004101 1597 tablet_replica.cc:333] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:29.004312 1597 raft_consensus.cc:2261] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:29.004483 1597 raft_consensus.cc:2290] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:29.004942 1492 tablet_bootstrap.cc:492] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:29.005107 1492 ts_tablet_manager.cc:1449] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20261118 08:54:29.006004 1492 raft_consensus.cc:360] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:29.006228 1492 raft_consensus.cc:386] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.006331 1492 raft_consensus.cc:748] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.006661 1492 consensus_queue.cc:261] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:29.008411 1492 ts_tablet_manager.cc:1480] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20261118 08:54:29.009193 1492 tablet_bootstrap.cc:492] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:29.012319 1492 tablet_bootstrap.cc:654] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:29.012897 1597 ts_tablet_manager.cc:1962] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:29.014358 1599 tablet_bootstrap.cc:492] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:29.014552 1599 ts_tablet_manager.cc:1449] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.001s
I20261118 08:54:29.014847 1492 tablet_bootstrap.cc:492] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:29.015009 1492 ts_tablet_manager.cc:1449] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20261118 08:54:29.015478 1599 raft_consensus.cc:360] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.015691 1599 raft_consensus.cc:386] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.015772 1599 raft_consensus.cc:748] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.015894 1492 raft_consensus.cc:360] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.016084 1492 raft_consensus.cc:386] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.016161 1492 raft_consensus.cc:748] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.016094 1599 consensus_queue.cc:261] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.016448 1492 consensus_queue.cc:261] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.016976 1492 ts_tablet_manager.cc:1480] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 08:54:29.017108 1598 ts_tablet_manager.cc:1975] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:29.017258 1598 log.cc:1199] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/8285b9829b7d4250a275063f36e1c93a
I20261118 08:54:29.017373 1492 tablet_bootstrap.cc:492] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:29.017697 1598 ts_tablet_manager.cc:1996] T 8285b9829b7d4250a275063f36e1c93a P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:29.019327 1492 tablet_bootstrap.cc:654] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:29.019726 1599 ts_tablet_manager.cc:1480] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 08:54:29.024278 1604 tablet_bootstrap.cc:492] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:29.024483 1604 ts_tablet_manager.cc:1449] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.007s
I20261118 08:54:29.025434 1604 raft_consensus.cc:360] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:29.025656 1604 raft_consensus.cc:386] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.025751 1604 raft_consensus.cc:748] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.026031 1604 consensus_queue.cc:261] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:29.026680 1604 ts_tablet_manager.cc:1480] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20261118 08:54:29.027009 1604 tablet_bootstrap.cc:492] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:29.029435 1604 tablet_bootstrap.cc:654] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:29.033851 1597 ts_tablet_manager.cc:1975] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:29.033988 1597 log.cc:1199] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/4a6c27574dfb45a1b736aad9c209669e
I20261118 08:54:29.034363 1597 ts_tablet_manager.cc:1996] T 4a6c27574dfb45a1b736aad9c209669e P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:29.036602 1597 tablet_replica.cc:333] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:29.036787 1597 raft_consensus.cc:2261] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:29.036991 1597 raft_consensus.cc:2290] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:29.037220 1604 tablet_bootstrap.cc:492] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:29.037427 1604 ts_tablet_manager.cc:1449] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20261118 08:54:29.037804 1597 ts_tablet_manager.cc:1962] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:29.038371 1604 raft_consensus.cc:360] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.038599 1604 raft_consensus.cc:386] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.038669 1604 raft_consensus.cc:748] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.039005 1604 consensus_queue.cc:261] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.040548 1604 ts_tablet_manager.cc:1480] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20261118 08:54:29.041715 1604 tablet_bootstrap.cc:492] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:29.043889 1604 tablet_bootstrap.cc:654] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:29.044816 1492 tablet_bootstrap.cc:492] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:29.044983 1492 ts_tablet_manager.cc:1449] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.009s
I20261118 08:54:29.045784 1604 tablet_bootstrap.cc:492] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:29.045761 1492 raft_consensus.cc:360] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.045914 1604 ts_tablet_manager.cc:1449] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20261118 08:54:29.045966 1492 raft_consensus.cc:386] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.046063 1492 raft_consensus.cc:748] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.046283 1492 consensus_queue.cc:261] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.047713 1604 raft_consensus.cc:360] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.047925 1604 raft_consensus.cc:386] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.048012 1604 raft_consensus.cc:748] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.048277 1604 consensus_queue.cc:261] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.050726 1604 ts_tablet_manager.cc:1480] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20261118 08:54:29.050971 1492 ts_tablet_manager.cc:1480] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20261118 08:54:29.053212 1345 raft_consensus.cc:499] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:29.053390 1345 raft_consensus.cc:521] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:29.053701 1597 ts_tablet_manager.cc:1975] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:29.053839 1597 log.cc:1199] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/8285b9829b7d4250a275063f36e1c93a
I20261118 08:54:29.054075 1345 leader_election.cc:290] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:29.054282 1597 ts_tablet_manager.cc:1996] T 8285b9829b7d4250a275063f36e1c93a P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:29.054986 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c02766f2f8d94f479e0ce2dc03567a7d" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:29.055193 938 raft_consensus.cc:2490] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:29.055348 1604 tablet_bootstrap.cc:492] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:29.055714 1012 leader_election.cc:304] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:29.056020 1345 raft_consensus.cc:2826] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:29.056099 1345 raft_consensus.cc:499] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:29.056185 1345 raft_consensus.cc:3096] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.056350 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c02766f2f8d94f479e0ce2dc03567a7d" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:29.056560 1211 raft_consensus.cc:2490] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 0.
I20261118 08:54:29.057531 1604 tablet_bootstrap.cc:654] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:29.057936 1597 tablet_replica.cc:333] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:29.058156 1597 raft_consensus.cc:2261] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:29.058753 1597 raft_consensus.cc:2290] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:29.058985 1345 raft_consensus.cc:521] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:29.059505 1597 ts_tablet_manager.cc:1962] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:29.060072 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c02766f2f8d94f479e0ce2dc03567a7d" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:29.060222 1345 leader_election.cc:290] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Requested vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:29.060278 1211 raft_consensus.cc:3096] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.063593 1211 raft_consensus.cc:2490] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:29.063670 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c02766f2f8d94f479e0ce2dc03567a7d" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:29.063874 938 raft_consensus.cc:3096] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.064239 1011 leader_election.cc:304] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters:
I20261118 08:54:29.064589 1345 raft_consensus.cc:2826] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:29.066946 938 raft_consensus.cc:2490] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
I20261118 08:54:29.073475 1597 ts_tablet_manager.cc:1975] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:29.073618 1597 log.cc:1199] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/8b0f9c9d6eef4b66aaab3bcafd810360
I20261118 08:54:29.073993 1597 ts_tablet_manager.cc:1996] T 8b0f9c9d6eef4b66aaab3bcafd810360 P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:29.077414 1604 tablet_bootstrap.cc:492] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:29.077587 1604 ts_tablet_manager.cc:1449] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.010s
I20261118 08:54:29.078325 1604 raft_consensus.cc:360] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.078516 1604 raft_consensus.cc:386] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:29.078576 1604 raft_consensus.cc:748] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:29.078835 1604 consensus_queue.cc:261] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.078892 1345 raft_consensus.cc:705] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Becoming Leader. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Running, Role: LEADER
I20261118 08:54:29.079208 1345 consensus_queue.cc:238] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:29.079319 1604 ts_tablet_manager.cc:1480] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 08:54:29.093987 1617 consensus_queue.cc:1049] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:29.103856 1617 consensus_queue.cc:1049] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:29.184353 1502 raft_consensus.cc:499] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:29.184571 1502 raft_consensus.cc:521] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.185498 1502 leader_election.cc:290] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:29.186062 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d7f4643e2be049d7b2e3249a40c6b2ac" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:29.186086 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d7f4643e2be049d7b2e3249a40c6b2ac" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:29.186312 938 raft_consensus.cc:2490] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:29.186321 1075 raft_consensus.cc:2490] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:29.186769 1147 leader_election.cc:304] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:29.187071 1502 raft_consensus.cc:2826] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:29.187189 1502 raft_consensus.cc:499] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:29.187291 1502 raft_consensus.cc:3096] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.190193 1502 raft_consensus.cc:521] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.190884 1502 leader_election.cc:290] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Requested vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:29.191450 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d7f4643e2be049d7b2e3249a40c6b2ac" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:29.191435 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d7f4643e2be049d7b2e3249a40c6b2ac" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:29.191670 1075 raft_consensus.cc:3096] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.191669 938 raft_consensus.cc:3096] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.194696 1075 raft_consensus.cc:2490] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:29.194698 938 raft_consensus.cc:2490] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:29.195295 1147 leader_election.cc:304] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:29.195685 1502 raft_consensus.cc:2826] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:29.196048 1502 raft_consensus.cc:705] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [term 1 LEADER]: Becoming Leader. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Running, Role: LEADER
I20261118 08:54:29.196377 1502 consensus_queue.cc:238] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.200181 1626 consensus_queue.cc:1049] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:29.202478 1502 consensus_queue.cc:1049] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 08:54:29.221094 1145 connection.cc:573] server connection from 127.0.0.1:59224 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:29.221416 871 connection.cc:573] server connection from 127.0.0.1:44738 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:29.221423 1010 connection.cc:573] server connection from 127.0.0.1:52358 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:29.221741 1264 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:29.226984 1567 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:29.225814 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.190:42341 (local address 127.0.72.129:42431)
1118 08:54:29.225973 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:29.225989 (+ 16us) client_negotiation.cc:175] Beginning negotiation
1118 08:54:29.226103 (+ 114us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 08:54:29.226952 (+ 849us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":63}
W20261118 08:54:29.227737 1631 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:29.223983 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.190:42341 (local address 127.0.0.1:54816)
1118 08:54:29.224406 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:29.224424 (+ 18us) client_negotiation.cc:175] Beginning negotiation
1118 08:54:29.224556 (+ 132us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 08:54:29.227699 (+ 3143us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":751,"threads_started":1}
W20261118 08:54:29.229310 1565 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:29.226188 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.190:42341 (local address 127.0.72.130:58141)
1118 08:54:29.226329 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:29.226345 (+ 16us) client_negotiation.cc:175] Beginning negotiation
1118 08:54:29.226451 (+ 106us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 08:54:29.229283 (+ 2832us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48}
W20261118 08:54:29.229801 1566 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:29.226581 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.190:42341 (local address 127.0.72.131:35307)
1118 08:54:29.226714 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:29.226729 (+ 15us) client_negotiation.cc:175] Beginning negotiation
1118 08:54:29.226829 (+ 100us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 08:54:29.229776 (+ 2947us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
I20261118 08:54:29.271831 1510 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:29.272025 1510 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.272631 1510 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:29.273303 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8d92c02e29724e80820a7114d8bbdf7a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:29.273275 1210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8d92c02e29724e80820a7114d8bbdf7a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:29.273514 1075 raft_consensus.cc:2490] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:29.273553 1210 raft_consensus.cc:2490] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:29.274156 873 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:29.274734 1510 raft_consensus.cc:2826] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:29.274843 1510 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:29.274933 1510 raft_consensus.cc:3096] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.276991 1510 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.277599 1510 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Requested vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:29.278119 1210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8d92c02e29724e80820a7114d8bbdf7a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:29.278316 1210 raft_consensus.cc:3096] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.278267 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "8d92c02e29724e80820a7114d8bbdf7a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:29.278494 1075 raft_consensus.cc:3096] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.280494 1210 raft_consensus.cc:2490] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:29.280887 873 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:29.281147 1510 raft_consensus.cc:2826] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:29.281301 1510 raft_consensus.cc:705] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Becoming Leader. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Running, Role: LEADER
I20261118 08:54:29.281626 1075 raft_consensus.cc:2490] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:29.282887 1510 consensus_queue.cc:238] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.287612 1560 consensus_queue.cc:1049] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:29.288010 1510 consensus_queue.cc:1049] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:29.368247 1510 raft_consensus.cc:499] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:29.368485 1510 raft_consensus.cc:521] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.369315 1510 leader_election.cc:290] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:29.369869 1210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "fc173583a9584741b28f2c0203504a3a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:29.369941 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "fc173583a9584741b28f2c0203504a3a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:29.370129 1210 raft_consensus.cc:2490] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:29.370223 1075 raft_consensus.cc:2490] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:29.370555 873 leader_election.cc:304] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:29.370820 1510 raft_consensus.cc:2826] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:29.370944 1510 raft_consensus.cc:499] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:29.371047 1510 raft_consensus.cc:3096] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.373263 1510 raft_consensus.cc:521] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.373885 1510 leader_election.cc:290] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Requested vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:29.374677 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "fc173583a9584741b28f2c0203504a3a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:29.374876 1075 raft_consensus.cc:3096] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.375541 1210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "fc173583a9584741b28f2c0203504a3a" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:29.375802 1210 raft_consensus.cc:3096] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:29.378446 1075 raft_consensus.cc:2490] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:29.378859 873 leader_election.cc:304] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:29.379009 1210 raft_consensus.cc:2490] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:29.379173 1510 raft_consensus.cc:2826] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:29.379305 1510 raft_consensus.cc:705] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Becoming Leader. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Running, Role: LEADER
I20261118 08:54:29.379662 1510 consensus_queue.cc:238] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:29.382709 1558 consensus_queue.cc:1049] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:29.384151 1558 consensus_queue.cc:1049] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:29.455065 1641 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:29.455246 1641 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:29.455938 1641 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:29.457999 657 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:29.458168 658 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:29.458282 658 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:29.458511 1641 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:29.542229 290 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task3eSafg/build/asan/bin/kudu pid 639
I20261118 08:54:29.543421 703 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 08:54:29.675859 290 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task3eSafg/build/asan/bin/kudu with pid 639
I20261118 08:54:30.336423 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--webserver_interface=127.0.72.189
--webserver_port=35929
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 08:54:30.680246 1653 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:30.680591 1653 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:30.680701 1653 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:30.689247 1653 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:30.689358 1653 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:30.689433 1653 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:30.689497 1653 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:30.699949 1653 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.72.189
--webserver_port=35929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:30.701694 1653 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:30.703778 1653 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:30.711786 1661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:30.711869 1662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:30.712631 1664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:30.713330 1653 server_base.cc:1061] running on GCE node
I20261118 08:54:30.714040 1653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:30.715507 1653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:30.716702 1653 hybrid_clock.cc:648] HybridClock initialized: now 1794992070716649 us; error 44 us; skew 500 ppm
I20261118 08:54:30.717003 1653 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:30.719137 1653 webserver.cc:577] Webserver started at http://127.0.72.189:35929/ using document root <none> and password file <none>
I20261118 08:54:30.719911 1653 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:30.720008 1653 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:30.726109 1653 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20261118 08:54:30.729140 1670 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:30.730661 1653 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20261118 08:54:30.730842 1653 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
uuid: "857a577b88ba46cc846ecd1a25a722ca"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:30.731626 1653 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:30.753530 1653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:30.754446 1653 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:30.754731 1653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:30.774755 1653 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.189:33939
I20261118 08:54:30.774757 1721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.189:33939 every 8 connection(s)
I20261118 08:54:30.776564 1653 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
I20261118 08:54:30.784112 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 1653
I20261118 08:54:30.784190 1722 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:30.788595 1722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap starting.
I20261118 08:54:30.800938 1722 log.cc:826] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:30.835799 1722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=28 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 08:54:30.836321 1722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap complete.
I20261118 08:54:30.841758 1722 raft_consensus.cc:360] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:30.842485 1722 raft_consensus.cc:748] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857a577b88ba46cc846ecd1a25a722ca, State: Initialized, Role: FOLLOWER
I20261118 08:54:30.843135 1722 consensus_queue.cc:261] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:30.843972 1727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:30.844249 1727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:30.845108 1722 sys_catalog.cc:565] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:30.856894 1741 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:30.857098 1741 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: loading cluster ID for follower catalog manager: success
I20261118 08:54:30.861529 1741 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: acquiring CA information for follower catalog manager: success
I20261118 08:54:30.864830 1741 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:30.866236 1653 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:54:30.870672 1653 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:30.921033 1258 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:30.921142 1123 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:30.925102 985 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:30.925268 1687 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:35731; Asking this server to re-register.
I20261118 08:54:30.925395 1686 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:42731; Asking this server to re-register.
I20261118 08:54:30.926259 1123 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:30.926622 1687 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:48349; Asking this server to re-register.
I20261118 08:54:30.927291 985 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:30.927832 1258 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:30.928539 1687 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:30.929653 1686 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:30.929838 1685 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:31.289635 1727 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:31.289908 1727 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:31.291962 1727 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:31.292603 1672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:31.294041 1672 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:31.294247 1671 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:31.294368 1671 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:31.294775 1727 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:31.902907 1754 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:31.903082 1754 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:31.903862 1754 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:31.906141 1672 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:31.906347 1671 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:31.906461 1671 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:31.906694 1754 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:32.628974 1770 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:32.629154 1770 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:32.629761 1770 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:32.631855 1671 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:32.632030 1672 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:32.632192 1672 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:32.632450 1770 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:32.972749 290 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task3eSafg/build/asan/bin/kudu pid 1653
I20261118 08:54:32.979182 1717 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 08:54:33.084019 290 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task3eSafg/build/asan/bin/kudu with pid 1653
I20261118 08:54:33.605788 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--webserver_interface=127.0.72.189
--webserver_port=35929
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 08:54:33.948292 1790 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:33.948655 1790 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:33.948773 1790 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:33.957535 1790 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:33.957633 1790 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:33.957687 1790 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:33.957742 1790 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:33.968124 1790 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.72.189
--webserver_port=35929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:33.969779 1790 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:33.971678 1790 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:33.980048 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:33.980067 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:33.980266 1802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:33.981252 1790 server_base.cc:1061] running on GCE node
I20261118 08:54:33.982055 1790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:33.983558 1790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:33.984758 1790 hybrid_clock.cc:648] HybridClock initialized: now 1794992073984701 us; error 50 us; skew 500 ppm
I20261118 08:54:33.985090 1790 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:33.987232 1790 webserver.cc:577] Webserver started at http://127.0.72.189:35929/ using document root <none> and password file <none>
I20261118 08:54:33.988013 1790 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:33.988123 1790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:33.993777 1790 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20261118 08:54:33.996807 1808 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:33.998427 1790 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261118 08:54:33.998595 1790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
uuid: "857a577b88ba46cc846ecd1a25a722ca"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:33.999312 1790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:34.014524 1790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:34.015448 1790 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:34.015741 1790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:34.034358 1790 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.189:33939
I20261118 08:54:34.034348 1859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.189:33939 every 8 connection(s)
I20261118 08:54:34.036114 1790 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
I20261118 08:54:34.042789 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 1790
I20261118 08:54:34.043184 1860 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:34.047022 1860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap starting.
I20261118 08:54:34.059885 1860 log.cc:826] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:34.091569 1860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=28 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 08:54:34.092113 1860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap complete.
I20261118 08:54:34.098269 1860 raft_consensus.cc:360] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:34.099370 1860 raft_consensus.cc:748] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857a577b88ba46cc846ecd1a25a722ca, State: Initialized, Role: FOLLOWER
I20261118 08:54:34.100063 1860 consensus_queue.cc:261] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:34.100914 1864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:34.101200 1864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:34.102216 1860 sys_catalog.cc:565] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:34.117811 1878 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:34.112875 1790 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:54:34.118331 1790 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:34.117982 1878 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: loading cluster ID for follower catalog manager: success
I20261118 08:54:34.122696 1878 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: acquiring CA information for follower catalog manager: success
I20261118 08:54:34.125782 1878 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:34.434473 985 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:34.437698 1825 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:35571; Asking this server to re-register.
I20261118 08:54:34.438459 985 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:34.440364 1825 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:34.450368 1123 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:34.451653 1258 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:34.452072 1825 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:34163; Asking this server to re-register.
I20261118 08:54:34.452872 1123 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:34.452898 1825 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:50553; Asking this server to re-register.
I20261118 08:54:34.453461 1258 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:34.454926 1825 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:34.455084 1824 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:34.703913 1887 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:34.704208 1887 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:34.706130 1887 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:34.707976 1809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:34.709331 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:34.709542 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:34.709676 1809 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:34.710012 1887 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:35.271229 1894 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:35.271445 1894 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:35.272192 1894 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:35.274032 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:35.274194 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:35.274315 1810 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:35.274534 1894 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:35.878988 1904 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:35.879158 1904 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:35.879787 1904 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:35.885272 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:35.885445 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:35.885576 1810 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:35.885815 1904 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:36.671113 1920 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:36.671316 1920 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:36.672034 1920 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:36.673853 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:36.674011 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:36.674132 1809 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:36.674335 1920 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20261118 08:54:37.498605 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--webserver_interface=127.0.72.190
--webserver_port=41231
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 08:54:37.840474 1936 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:37.840798 1936 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:37.840876 1936 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:37.849210 1936 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:37.849303 1936 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:37.849356 1936 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:37.849403 1936 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:37.860102 1936 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.72.190
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:37.861647 1936 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:37.863332 1936 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:37.871068 1941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:37.871091 1942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:37.871731 1944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:37.872495 1936 server_base.cc:1061] running on GCE node
I20261118 08:54:37.873270 1936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:37.874717 1936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:37.875929 1936 hybrid_clock.cc:648] HybridClock initialized: now 1794992077875876 us; error 33 us; skew 500 ppm
I20261118 08:54:37.876283 1936 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:37.878607 1936 webserver.cc:577] Webserver started at http://127.0.72.190:41231/ using document root <none> and password file <none>
I20261118 08:54:37.879313 1936 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:37.879446 1936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:37.885092 1936 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20261118 08:54:37.888119 1950 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:37.889607 1936 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261118 08:54:37.889775 1936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:37.890481 1936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:37.909832 1936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:37.910746 1936 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:37.911073 1936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:37.932791 1936 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.190:42341
I20261118 08:54:37.932793 2005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.190:42341 every 8 connection(s)
I20261118 08:54:37.934542 1936 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
I20261118 08:54:37.936503 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 1936
I20261118 08:54:37.943734 2006 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:37.949684 2006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap starting.
I20261118 08:54:37.965703 2006 log.cc:826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:37.968868 984 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:38.003865 2006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=28 ignored=0} mutations{seen=29 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 08:54:38.004470 2006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap complete.
I20261118 08:54:38.010102 2006 raft_consensus.cc:360] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:38.010878 2006 raft_consensus.cc:748] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Initialized, Role: FOLLOWER
I20261118 08:54:38.011572 2006 consensus_queue.cc:261] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:38.012372 2013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:38.012611 2013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:38.013458 2006 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:38.028909 2027 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:38.029009 2027 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: loading cluster ID for follower catalog manager: success
I20261118 08:54:38.033143 2027 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: acquiring CA information for follower catalog manager: success
I20261118 08:54:38.036168 2027 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:38.048614 1936 master_runner.cc:187] Error getting master registration for 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111),
I20261118 08:54:38.052482 1936 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:38.431475 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:38.434144 1967 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:36541; Asking this server to re-register.
I20261118 08:54:38.434937 1122 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:38.437098 1967 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:38.438434 1257 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:38.440115 1967 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:47025; Asking this server to re-register.
I20261118 08:54:38.440724 1257 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:38.441984 1967 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:38.474819 1967 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:60377; Asking this server to re-register.
I20261118 08:54:38.475494 984 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:38.476765 1967 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:38.547330 2037 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:38.547628 2037 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:38.549396 2037 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:38.550100 1835 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
W20261118 08:54:38.550227 1951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:38.550536 1835 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 3.
I20261118 08:54:38.551163 1952 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters:
I20261118 08:54:38.551549 2037 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20261118 08:54:38.551712 2037 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20261118 08:54:38.551739 1951 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:38.551821 2037 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 3 FOLLOWER]: Advancing to term 4
I20261118 08:54:38.555420 2037 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:38.556025 2037 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 4 election: Requested vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:38.556504 1835 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca"
I20261118 08:54:38.556701 1835 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 3 FOLLOWER]: Advancing to term 4
W20261118 08:54:38.557291 1951 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:38.560171 1835 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 4.
I20261118 08:54:38.560583 1952 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1
I20261118 08:54:38.560910 2037 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Leader election won for term 4
I20261118 08:54:38.561303 2037 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 LEADER]: Becoming Leader. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Running, Role: LEADER
I20261118 08:54:38.561751 2037 consensus_queue.cc:238] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:38.564471 2040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 4 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:38.564848 2040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
W20261118 08:54:38.566103 1951 consensus_peers.cc:597] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 -> Peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Couldn't send request to peer 6c7d9f52503f41e4bafe2821e6fd6af1. Status: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:38.566125 1835 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Refusing update from remote peer 78d4026bc230421ea10b8c68a9ef7871: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 4 index: 41. (index mismatch)
I20261118 08:54:38.566212 2044 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 08:54:38.566967 2040 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20261118 08:54:38.569252 2044 catalog_manager.cc:694] Loaded metadata for table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]
I20261118 08:54:38.569825 2044 catalog_manager.cc:694] Loaded metadata for table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]
I20261118 08:54:38.570199 2044 catalog_manager.cc:694] Loaded metadata for table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]
I20261118 08:54:38.570542 2044 catalog_manager.cc:694] Loaded metadata for table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]
I20261118 08:54:38.570642 2045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 4 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:38.570847 2044 catalog_manager.cc:694] Loaded metadata for table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]
I20261118 08:54:38.570859 2045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:38.571182 2044 catalog_manager.cc:694] Loaded metadata for table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]
I20261118 08:54:38.571520 2044 catalog_manager.cc:694] Loaded metadata for table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]
I20261118 08:54:38.571931 2044 catalog_manager.cc:694] Loaded metadata for table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]
I20261118 08:54:38.572305 2044 catalog_manager.cc:694] Loaded metadata for table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]
I20261118 08:54:38.573462 2042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:38.573683 2042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:38.574127 2045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:38.574325 2045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:38.575042 2044 tablet_loader.cc:96] loaded metadata for tablet 39e7203496e0458687be52c3fadd15ea (table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30])
I20261118 08:54:38.575328 2044 tablet_loader.cc:96] loaded metadata for tablet 3e292319f0c44e34b49ae6d579df2ecb (table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a])
I20261118 08:54:38.575809 2044 tablet_loader.cc:96] loaded metadata for tablet 543333f1b0df4517a424aa536902aa29 (table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a])
I20261118 08:54:38.576488 2044 tablet_loader.cc:96] loaded metadata for tablet 8d92c02e29724e80820a7114d8bbdf7a (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2])
I20261118 08:54:38.576895 2044 tablet_loader.cc:96] loaded metadata for tablet c02766f2f8d94f479e0ce2dc03567a7d (table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8])
I20261118 08:54:38.577153 2044 tablet_loader.cc:96] loaded metadata for tablet d546b7f20a7f4d3b9e5612080df04be6 (table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70])
I20261118 08:54:38.577440 2044 tablet_loader.cc:96] loaded metadata for tablet d7f4643e2be049d7b2e3249a40c6b2ac (table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b])
I20261118 08:54:38.577713 2044 tablet_loader.cc:96] loaded metadata for tablet d92e535f2bf44815b6c003b56dd25b79 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957])
I20261118 08:54:38.577973 2044 tablet_loader.cc:96] loaded metadata for tablet fc173583a9584741b28f2c0203504a3a (table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7])
I20261118 08:54:38.578210 2044 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 08:54:38.579172 2044 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:38.579246 2044 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 08:54:38.580497 2044 catalog_manager.cc:1555] Loading token signing keys...
I20261118 08:54:38.581327 2044 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Loaded TSK: 0
I20261118 08:54:38.582141 2044 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 08:54:38.582975 2044 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 08:54:38.583799 2044 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 08:54:38.656008 1967 master_service.cc:965] ReplaceTablet: received request to replace tablet d7f4643e2be049d7b2e3249a40c6b2ac from {username='slave'} at 127.0.0.1:43154
I20261118 08:54:38.656168 1965 master_service.cc:965] ReplaceTablet: received request to replace tablet 8d92c02e29724e80820a7114d8bbdf7a from {username='slave'} at 127.0.0.1:43154
I20261118 08:54:38.656546 1963 master_service.cc:965] ReplaceTablet: received request to replace tablet 39e7203496e0458687be52c3fadd15ea from {username='slave'} at 127.0.0.1:43154
I20261118 08:54:38.657352 1960 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43154:
table { table_name: "default.Table_41d62c0cf6b54938b8e1770b859e56e8" } new_table_name: "default.Table_eefa15fa899c4b1e9162efce0db7e4af" modify_external_catalogs: true
I20261118 08:54:38.657423 1961 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43154:
table { table_name: "default.Table_5f1fb0758cf1413eaed1d8b4717aa902" } new_table_name: "default.Table_d9954c97f0054586ab7cf9e5e98b3fa4" modify_external_catalogs: true
I20261118 08:54:38.657898 1959 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43154:
table { table_name: "default.Table_bc1f66261ac64a87912e19a32ce0ca44" } new_table_name: "default.Table_7fe6a97ebd294ce68601eaf9f252d0ee" modify_external_catalogs: true
I20261118 08:54:38.658116 1964 master_service.cc:965] ReplaceTablet: received request to replace tablet 8d92c02e29724e80820a7114d8bbdf7a from {username='slave'} at 127.0.0.1:43154
I20261118 08:54:38.656033 1966 master_service.cc:965] ReplaceTablet: received request to replace tablet d92e535f2bf44815b6c003b56dd25b79 from {username='slave'} at 127.0.0.1:43154
I20261118 08:54:38.658931 1958 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43154:
name: "default.Table_0327d29a2f19450b872c54c554d7654e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:38.660022 1962 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43154:
name: "default.Table_8f6feb81c1374d42bc1e0acb856b75cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20261118 08:54:38.667511 2026 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_bc1f66261ac64a87912e19a32ce0ca44, message={"server":"","servicePrincipal":"","db":"default","table":"table_bc1f66261ac64a87912e19a32ce0ca44","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_bc1f66261ac64a87912e19a32ce0ca44\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_bc1f66261ac64a87912e19a32ce0ca44\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_bc1f66261ac64a87912e19a32ce0ca44\",\"kudu.table_id\":\"4fcf984156584e30aac26283f71adda2\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_8b08d3e4ed824f1680377a6d5e7518a8, message={"server":"","servicePrincipal":"","db":"default","table":"table_8b08d3e4ed824f1680377a6d5e7518a8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_8b08d3e4ed824f1680377a6d5e7518a8\",\"kudu.table_id\":\"b42eb739941947f8b7d79fddd923eb52\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20261118 08:54:38.669618 1966 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:38.669919 1967 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet d7f4643e2be049d7b2e3249a40c6b2ac
@ 0x7fe33e914d5d google::LogMessage::Fail() at ??:0
I20261118 08:54:38.671886 1966 catalog_manager.cc:6936] ReplaceTablet: tablet d92e535f2bf44815b6c003b56dd25b79 deleted and replaced by tablet 86bf01af75674f5491b024ac43a063b0
I20261118 08:54:38.672379 1966 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43154:
name: "default.Table_82d1541f550a4d5482cff56810c97a22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fe33e918b23 google::LogMessage::SendToLog() at ??:0
I20261118 08:54:38.674481 1967 catalog_manager.cc:6936] ReplaceTablet: tablet d7f4643e2be049d7b2e3249a40c6b2ac deleted and replaced by tablet 748947f292334e7a97abfa599634006d
I20261118 08:54:38.674811 1967 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43154:
table { table_name: "default.Table_1ee6cacf31104df284b253928039759b" } new_table_name: "default.Table_29f622723bbb4a47970de8920bf7d580" modify_external_catalogs: true
@ 0x7fe33e91475c google::LogMessage::Flush() at ??:0
@ 0x7fe33e9149a1 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe353d949a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fe353d93251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fe353d9aecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 08:54:38.700053 1965 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:54:38.700685 1965 catalog_manager.cc:6936] ReplaceTablet: tablet 8d92c02e29724e80820a7114d8bbdf7a deleted and replaced by tablet 3c8c4245c7ac4d8ca2e7fa521b6d49d3
I20261118 08:54:38.701074 1965 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43154:
name: "default.Table_903cefd8f2a74c89be4bf41b84d26ba4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:38.701952 1964 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43154:
name: "default.Table_52bba9536ae942a59235d8087d61576b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:38.702536 1963 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 39e7203496e0458687be52c3fadd15ea
I20261118 08:54:38.703162 1963 catalog_manager.cc:6936] ReplaceTablet: tablet 39e7203496e0458687be52c3fadd15ea deleted and replaced by tablet 50744c2d59124a22ac0a76f2f8e60974
@ 0x7fe353d9ad49 std::_Function_handler<>::_M_invoke() at ??:0
I20261118 08:54:38.703526 1963 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43154:
name: "default.Table_bfd124a61a904e60bc99bee8716f64cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
@ 0x7fe33fd5ddc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe3434386db start_thread at ??:0
@ 0x7fe33bea861f clone at ??:0
I20261118 08:54:38.712870 916 tablet_service.cc:1547] Processing DeleteTablet for tablet d92e535f2bf44815b6c003b56dd25b79 with delete_type TABLET_DATA_DELETED (replaced by tablet 86bf01af75674f5491b024ac43a063b0) from {username='slave'} at 127.0.0.1:37254
I20261118 08:54:38.712965 917 tablet_service.cc:1547] Processing DeleteTablet for tablet d7f4643e2be049d7b2e3249a40c6b2ac with delete_type TABLET_DATA_DELETED (replaced by tablet 748947f292334e7a97abfa599634006d) from {username='slave'} at 127.0.0.1:37254
I20261118 08:54:38.713096 918 tablet_service.cc:1547] Processing DeleteTablet for tablet 8d92c02e29724e80820a7114d8bbdf7a with delete_type TABLET_DATA_DELETED (replaced by tablet 3c8c4245c7ac4d8ca2e7fa521b6d49d3) from {username='slave'} at 127.0.0.1:37254
I20261118 08:54:38.713243 915 tablet_service.cc:1547] Processing DeleteTablet for tablet 39e7203496e0458687be52c3fadd15ea with delete_type TABLET_DATA_DELETED (replaced by tablet 50744c2d59124a22ac0a76f2f8e60974) from {username='slave'} at 127.0.0.1:37254
I20261118 08:54:38.715421 2061 tablet_replica.cc:333] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:38.715626 2061 raft_consensus.cc:2261] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:38.715602 1054 tablet_service.cc:1547] Processing DeleteTablet for tablet d7f4643e2be049d7b2e3249a40c6b2ac with delete_type TABLET_DATA_DELETED (replaced by tablet 748947f292334e7a97abfa599634006d) from {username='slave'} at 127.0.0.1:42700
I20261118 08:54:38.715698 1055 tablet_service.cc:1547] Processing DeleteTablet for tablet d92e535f2bf44815b6c003b56dd25b79 with delete_type TABLET_DATA_DELETED (replaced by tablet 86bf01af75674f5491b024ac43a063b0) from {username='slave'} at 127.0.0.1:42700
I20261118 08:54:38.715850 2061 raft_consensus.cc:2290] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.716259 2062 tablet_replica.cc:333] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:38.716382 1054 tablet_service.cc:1547] Processing DeleteTablet for tablet 8d92c02e29724e80820a7114d8bbdf7a with delete_type TABLET_DATA_DELETED (replaced by tablet 3c8c4245c7ac4d8ca2e7fa521b6d49d3) from {username='slave'} at 127.0.0.1:42700
I20261118 08:54:38.716439 2061 ts_tablet_manager.cc:1962] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.716457 2062 raft_consensus.cc:2261] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:38.716691 2062 raft_consensus.cc:2290] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.716753 1054 tablet_service.cc:1547] Processing DeleteTablet for tablet 39e7203496e0458687be52c3fadd15ea with delete_type TABLET_DATA_DELETED (replaced by tablet 50744c2d59124a22ac0a76f2f8e60974) from {username='slave'} at 127.0.0.1:42700
I20261118 08:54:38.717181 2062 ts_tablet_manager.cc:1962] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.721899 2062 ts_tablet_manager.cc:1975] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.722026 2062 log.cc:1199] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/d7f4643e2be049d7b2e3249a40c6b2ac
I20261118 08:54:38.722479 2062 ts_tablet_manager.cc:1996] T d7f4643e2be049d7b2e3249a40c6b2ac P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:38.723043 2061 ts_tablet_manager.cc:1975] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.723160 2061 log.cc:1199] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/d7f4643e2be049d7b2e3249a40c6b2ac
I20261118 08:54:38.723651 2061 ts_tablet_manager.cc:1996] T d7f4643e2be049d7b2e3249a40c6b2ac P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:38.723691 2062 tablet_replica.cc:333] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:38.723891 2062 raft_consensus.cc:2261] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:38.724050 2062 raft_consensus.cc:2290] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.724591 2062 ts_tablet_manager.cc:1962] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.724761 2061 tablet_replica.cc:333] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:38.724941 2061 raft_consensus.cc:2261] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:38.725240 2061 raft_consensus.cc:2290] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.725767 2061 ts_tablet_manager.cc:1962] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.731061 2062 ts_tablet_manager.cc:1975] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.731192 2062 log.cc:1199] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:38.731587 2062 ts_tablet_manager.cc:1996] T d92e535f2bf44815b6c003b56dd25b79 P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:38.732056 2061 ts_tablet_manager.cc:1975] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.732203 2061 log.cc:1199] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:38.732549 2061 ts_tablet_manager.cc:1996] T d92e535f2bf44815b6c003b56dd25b79 P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:38.732853 2062 tablet_replica.cc:333] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:38.733029 2062 raft_consensus.cc:2261] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:38.733203 2062 raft_consensus.cc:2290] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.733735 2061 tablet_replica.cc:333] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:38.733758 2062 ts_tablet_manager.cc:1962] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.733911 2061 raft_consensus.cc:2261] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:38.734229 2061 raft_consensus.cc:2290] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.734766 2061 ts_tablet_manager.cc:1962] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.740310 2062 ts_tablet_manager.cc:1975] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.740425 2062 log.cc:1199] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:54:38.740833 2062 ts_tablet_manager.cc:1996] T 8d92c02e29724e80820a7114d8bbdf7a P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:38.741091 2061 ts_tablet_manager.cc:1975] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.741205 2061 log.cc:1199] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:54:38.741598 2061 ts_tablet_manager.cc:1996] T 8d92c02e29724e80820a7114d8bbdf7a P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:38.742005 2062 tablet_replica.cc:333] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:38.742197 2062 raft_consensus.cc:2261] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:38.742363 2062 raft_consensus.cc:2290] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.742674 2061 tablet_replica.cc:333] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:38.742874 2061 raft_consensus.cc:2261] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:38.742931 2062 ts_tablet_manager.cc:1962] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.743027 2061 raft_consensus.cc:2290] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:38.743607 2061 ts_tablet_manager.cc:1962] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:38.749228 2062 ts_tablet_manager.cc:1975] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.749365 2062 log.cc:1199] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/39e7203496e0458687be52c3fadd15ea
I20261118 08:54:38.749732 2062 ts_tablet_manager.cc:1996] T 39e7203496e0458687be52c3fadd15ea P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:38.749951 2061 ts_tablet_manager.cc:1975] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:38.750082 2061 log.cc:1199] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/39e7203496e0458687be52c3fadd15ea
I20261118 08:54:38.750525 2061 ts_tablet_manager.cc:1996] T 39e7203496e0458687be52c3fadd15ea P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
W20261118 08:54:38.883275 1012 connection.cc:573] server connection from 127.0.0.1:42700 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:38.883280 873 connection.cc:573] server connection from 127.0.0.1:37254 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:38.883319 1809 connection.cc:573] server connection from 127.0.0.1:41760 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:38.883894 1264 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:38.884480 1264 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.72.190:42341)
W20261118 08:54:38.884816 1264 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:38.884991 1264 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:38.885178 1264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:38.927958 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 1: this message will repeat every 5th retry.
W20261118 08:54:39.090116 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:39.119756 1790 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:54:39.119920 1790 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:39.211485 2076 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:39.211715 2076 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:39.212397 2076 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:39.214368 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:39.214526 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:39.214643 1809 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:39.214879 2076 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20261118 08:54:39.309316 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 1: this message will repeat every 5th retry.
W20261118 08:54:39.456259 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:39.764822 2079 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:39.765025 2079 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:39.765702 2079 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:39.766674 1810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20261118 08:54:39.767710 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:39.767871 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:39.767995 1810 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:39.768234 2079 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20261118 08:54:40.776459 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--webserver_interface=127.0.72.190
--webserver_port=41231
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 08:54:40.799005 2092 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:40.799183 2092 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:40.799950 2092 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:40.802079 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:40.802532 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:40.802661 1809 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:40.802950 2092 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20261118 08:54:41.126612 2091 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:41.126947 2091 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:41.127067 2091 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:41.136092 2091 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:41.136204 2091 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:41.136276 2091 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:41.136336 2091 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:41.146843 2091 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.72.190
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:41.148597 2091 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:41.150419 2091 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:41.158424 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:41.158455 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:41.159643 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:41.159700 2091 server_base.cc:1061] running on GCE node
I20261118 08:54:41.160454 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:41.161831 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:41.163043 2091 hybrid_clock.cc:648] HybridClock initialized: now 1794992081163002 us; error 30 us; skew 500 ppm
I20261118 08:54:41.163507 2091 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:41.165699 2091 webserver.cc:577] Webserver started at http://127.0.72.190:41231/ using document root <none> and password file <none>
I20261118 08:54:41.166421 2091 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:41.166548 2091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:41.172041 2091 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20261118 08:54:41.174877 2111 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:41.176355 2091 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261118 08:54:41.176560 2091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:41.177266 2091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:41.205526 2091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:41.206455 2091 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:41.206732 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:41.225314 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.190:42341
I20261118 08:54:41.225338 2162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.190:42341 every 8 connection(s)
I20261118 08:54:41.227650 2091 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
I20261118 08:54:41.233944 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 2091
I20261118 08:54:41.236660 2163 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:41.240811 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap starting.
I20261118 08:54:41.246886 1257 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:41.256824 2163 log.cc:826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:41.291499 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap replayed 1/1 log segments. Stats: ops{read=46 overwritten=0 applied=45 ignored=0} inserts{seen=32 ignored=0} mutations{seen=33 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20261118 08:54:41.292163 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap complete.
I20261118 08:54:41.297416 2163 raft_consensus.cc:360] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:41.299978 2163 raft_consensus.cc:748] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Initialized, Role: FOLLOWER
I20261118 08:54:41.300644 2163 consensus_queue.cc:261] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 4.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:41.301347 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:41.301606 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:41.302492 2163 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:41.322429 2184 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:41.322548 2184 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: loading cluster ID for follower catalog manager: success
I20261118 08:54:41.333088 2184 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: acquiring CA information for follower catalog manager: success
I20261118 08:54:41.338531 2184 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:41.344064 2085 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:41.344270 2085 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:41.345038 2085 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:54:41.346235 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:54:41.346608 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:41.346745 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:41.347000 2085 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:41.347980 2091 master_runner.cc:187] Error getting master registration for 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111),
I20261118 08:54:41.353579 2091 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:41.451195 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:41.453548 2128 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:35739; Asking this server to re-register.
I20261118 08:54:41.454272 1122 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:41.456065 2128 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:41.490257 984 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:41.491521 2128 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:36255; Asking this server to re-register.
I20261118 08:54:41.492142 984 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:41.493414 2128 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:41.619026 2085 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:41.619220 2085 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:41.619940 2085 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:54:41.620870 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:54:41.621011 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:54:41.621135 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:41.621389 2085 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:41.752985 2127 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:38145; Asking this server to re-register.
I20261118 08:54:41.753643 1257 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:41.754921 2127 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:41.785933 2170 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:41.786216 2170 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:41.788117 2170 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:41.788689 1835 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 5 candidate_status { last_received { term: 4 index: 46 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
W20261118 08:54:41.788769 2112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:41.788897 1835 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 4.
I20261118 08:54:41.789543 2113 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters:
I20261118 08:54:41.789920 2170 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20261118 08:54:41.790056 2170 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:41.790176 2170 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 4 FOLLOWER]: Advancing to term 5
W20261118 08:54:41.790189 2112 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:41.793629 2170 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:41.794229 2170 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 5 election: Requested vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:41.794701 1835 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 5 candidate_status { last_received { term: 4 index: 46 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca"
I20261118 08:54:41.794909 1835 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 4 FOLLOWER]: Advancing to term 5
W20261118 08:54:41.795624 2112 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:41.797029 1835 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 5.
I20261118 08:54:41.797428 2113 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1
I20261118 08:54:41.797689 2170 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Leader election won for term 5
I20261118 08:54:41.798125 2170 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 LEADER]: Becoming Leader. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Running, Role: LEADER
I20261118 08:54:41.798553 2170 consensus_queue.cc:238] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 4.46, Last appended by leader: 46, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:41.801198 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 5 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:41.801499 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:41.802472 1835 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Refusing update from remote peer 78d4026bc230421ea10b8c68a9ef7871: Log matching property violated. Preceding OpId in replica: term: 4 index: 46. Preceding OpId from leader: term: 5 index: 47. (index mismatch)
W20261118 08:54:41.803197 2112 consensus_peers.cc:597] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 -> Peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Couldn't send request to peer 6c7d9f52503f41e4bafe2821e6fd6af1. Status: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:41.803460 2190 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 45, Time since last communication: 0.000s
I20261118 08:54:41.805589 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 5 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:41.805917 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:41.816071 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:41.816321 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:41.817168 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:41.817390 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:41.818313 2194 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 08:54:41.820760 2194 catalog_manager.cc:694] Loaded metadata for table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]
I20261118 08:54:41.821252 2194 catalog_manager.cc:694] Loaded metadata for table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]
I20261118 08:54:41.821573 2194 catalog_manager.cc:694] Loaded metadata for table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]
I20261118 08:54:41.821861 2194 catalog_manager.cc:694] Loaded metadata for table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]
I20261118 08:54:41.822126 2194 catalog_manager.cc:694] Loaded metadata for table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]
I20261118 08:54:41.822499 2194 catalog_manager.cc:694] Loaded metadata for table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]
I20261118 08:54:41.822829 2194 catalog_manager.cc:694] Loaded metadata for table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]
I20261118 08:54:41.823119 2194 catalog_manager.cc:694] Loaded metadata for table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]
I20261118 08:54:41.823454 2194 catalog_manager.cc:694] Loaded metadata for table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]
I20261118 08:54:41.825683 2194 tablet_loader.cc:96] loaded metadata for tablet 3c8c4245c7ac4d8ca2e7fa521b6d49d3 (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2])
I20261118 08:54:41.826037 2194 tablet_loader.cc:96] loaded metadata for tablet 3e292319f0c44e34b49ae6d579df2ecb (table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a])
I20261118 08:54:41.826409 2194 tablet_loader.cc:96] loaded metadata for tablet 50744c2d59124a22ac0a76f2f8e60974 (table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30])
I20261118 08:54:41.826645 2194 tablet_loader.cc:96] loaded metadata for tablet 543333f1b0df4517a424aa536902aa29 (table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a])
I20261118 08:54:41.827014 2194 tablet_loader.cc:96] loaded metadata for tablet 748947f292334e7a97abfa599634006d (table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b])
I20261118 08:54:41.827411 2194 tablet_loader.cc:96] loaded metadata for tablet 86bf01af75674f5491b024ac43a063b0 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957])
I20261118 08:54:41.828136 2194 tablet_loader.cc:96] loaded metadata for tablet c02766f2f8d94f479e0ce2dc03567a7d (table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8])
I20261118 08:54:41.828378 2194 tablet_loader.cc:96] loaded metadata for tablet d546b7f20a7f4d3b9e5612080df04be6 (table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70])
I20261118 08:54:41.829104 2194 tablet_loader.cc:96] loaded metadata for tablet fc173583a9584741b28f2c0203504a3a (table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7])
I20261118 08:54:41.829449 2194 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 08:54:41.830688 2194 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:41.830760 2194 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 08:54:41.832032 2194 catalog_manager.cc:1555] Loading token signing keys...
I20261118 08:54:41.832919 2194 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Loaded TSK: 0
I20261118 08:54:41.833741 2194 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 08:54:41.834811 2194 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 08:54:41.835723 2194 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 08:54:41.909436 2127 master_service.cc:965] ReplaceTablet: received request to replace tablet fc173583a9584741b28f2c0203504a3a from {username='slave'} at 127.0.0.1:43814
I20261118 08:54:41.909548 2128 master_service.cc:965] ReplaceTablet: received request to replace tablet 3e292319f0c44e34b49ae6d579df2ecb from {username='slave'} at 127.0.0.1:43814
I20261118 08:54:41.910156 2126 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_ef26d2c593994e118ff3d01a0f5a9a1d" } new_table_name: "default.Table_0d932fab32a1414582d1f8370b6e2b23" modify_external_catalogs: true
I20261118 08:54:41.912067 2119 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff" } modify_external_catalogs: true
I20261118 08:54:41.914013 2119 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff" } modify_external_catalogs: true
I20261118 08:54:41.910920 2121 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43814:
name: "default.Table_3655f04cfdf84b26839c6e76aac6ebbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:41.913192 2120 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43814:
name: "default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:41.911432 2122 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43814:
name: "default.Table_738b3d8aface40768cac1e7b01d47046"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:41.913676 2123 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_294f03c6875c4da0845888273ea80a2a" } modify_external_catalogs: true
I20261118 08:54:41.913538 2125 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
I20261118 08:54:41.913583 2124 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43814:
name: "default.Table_b5863ec6b7e144f58594f23a0634bf0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:41.915833 2123 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_294f03c6875c4da0845888273ea80a2a" } modify_external_catalogs: true
I20261118 08:54:41.916203 2125 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
F20261118 08:54:41.920807 2183 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_bc1f66261ac64a87912e19a32ce0ca44, message={"server":"","servicePrincipal":"","db":"default","table":"table_bc1f66261ac64a87912e19a32ce0ca44","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_bc1f66261ac64a87912e19a32ce0ca44\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_bc1f66261ac64a87912e19a32ce0ca44\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_bc1f66261ac64a87912e19a32ce0ca44\",\"kudu.table_id\":\"4fcf984156584e30aac26283f71adda2\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_8b08d3e4ed824f1680377a6d5e7518a8, message={"server":"","servicePrincipal":"","db":"default","table":"table_8b08d3e4ed824f1680377a6d5e7518a8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_8b08d3e4ed824f1680377a6d5e7518a8\",\"kudu.table_id\":\"b42eb739941947f8b7d79fddd923eb52\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7f99eefd1d5d google::LogMessage::Fail() at ??:0
@ 0x7f99eefd5b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f99eefd175c google::LogMessage::Flush() at ??:0
I20261118 08:54:41.926334 2128 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 3e292319f0c44e34b49ae6d579df2ecb
@ 0x7f99eefd19a1 google::LogMessage::~LogMessage() at ??:0
I20261118 08:54:41.931813 2128 catalog_manager.cc:6936] ReplaceTablet: tablet 3e292319f0c44e34b49ae6d579df2ecb deleted and replaced by tablet 368ffd5a7d724f47abeea8f223415eac
I20261118 08:54:41.932560 2128 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43814:
name: "default.Table_8f6feb81c1374d42bc1e0acb856b75cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f9a044519a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 08:54:41.937314 917 tablet_service.cc:1547] Processing DeleteTablet for tablet 3e292319f0c44e34b49ae6d579df2ecb with delete_type TABLET_DATA_DELETED (replaced by tablet 368ffd5a7d724f47abeea8f223415eac) from {username='slave'} at 127.0.0.1:37256
@ 0x7f9a04450251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 08:54:41.944357 2127 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet fc173583a9584741b28f2c0203504a3a
I20261118 08:54:41.944939 2127 catalog_manager.cc:6936] ReplaceTablet: tablet fc173583a9584741b28f2c0203504a3a deleted and replaced by tablet d2ce64bc39634b5882b1beaa68e66cbe
I20261118 08:54:41.945310 2127 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "default.Table_5f1fb0758cf1413eaed1d8b4717aa902" } new_table_name: "default.Table_d9954c97f0054586ab7cf9e5e98b3fa4" modify_external_catalogs: true
@ 0x7f9a04457ecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 08:54:41.945468 915 tablet_service.cc:1547] Processing DeleteTablet for tablet fc173583a9584741b28f2c0203504a3a with delete_type TABLET_DATA_DELETED (replaced by tablet d2ce64bc39634b5882b1beaa68e66cbe) from {username='slave'} at 127.0.0.1:37256
@ 0x7f9a04457d49 std::_Function_handler<>::_M_invoke() at ??:0
I20261118 08:54:41.950743 2210 tablet_replica.cc:333] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:41.950951 2210 raft_consensus.cc:2261] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Raft consensus shutting down.
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 08:54:41.951332 2210 raft_consensus.cc:2290] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:41.952090 2210 ts_tablet_manager.cc:1962] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f99f041adc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f99f3af56db start_thread at ??:0
@ 0x7f99ec56561f clone at ??:0
I20261118 08:54:41.962895 2210 ts_tablet_manager.cc:1975] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:41.963006 2210 log.cc:1199] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/3e292319f0c44e34b49ae6d579df2ecb
I20261118 08:54:41.963457 2210 ts_tablet_manager.cc:1996] T 3e292319f0c44e34b49ae6d579df2ecb P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:41.964551 2210 tablet_replica.cc:333] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:41.964756 2210 raft_consensus.cc:2261] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:41.965077 2210 raft_consensus.cc:2290] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:41.965689 2210 ts_tablet_manager.cc:1962] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:41.970599 2210 ts_tablet_manager.cc:1975] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:41.970714 2210 log.cc:1199] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/fc173583a9584741b28f2c0203504a3a
I20261118 08:54:41.970952 2210 ts_tablet_manager.cc:1996] T fc173583a9584741b28f2c0203504a3a P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
W20261118 08:54:42.124670 873 connection.cc:573] server connection from 127.0.0.1:37256 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:42.124761 1809 connection.cc:573] server connection from 127.0.0.1:41764 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:42.125068 872 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:42.125095 1010 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:42.127063 1264 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.72.190:42341)
W20261118 08:54:42.127530 1264 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:42.127703 1264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:42.127985 1264 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.72.190:42341)
I20261118 08:54:42.459189 2225 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:42.459452 2225 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:42.460191 2225 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:42.462214 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:42.462415 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:42.462533 1809 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:42.462795 2225 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 08:54:43.102944 2235 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:43.103153 2235 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:43.103906 2235 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:43.105993 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:43.106249 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:43.106382 1810 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:43.106643 2235 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 08:54:43.905901 2244 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:43.906155 2244 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:43.906924 2244 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:43.908864 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:43.909019 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:43.909200 1809 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:43.909490 2244 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20261118 08:54:44.121034 1790 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:54:44.121227 1790 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:44.131490 2204 raft_consensus.cc:499] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:44.131685 2204 raft_consensus.cc:521] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:44.132512 2204 leader_election.cc:290] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:44.133042 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "3e292319f0c44e34b49ae6d579df2ecb" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
W20261118 08:54:44.133451 1012 leader_election.cc:343] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb
I20261118 08:54:44.133659 1012 leader_election.cc:304] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:44.133914 2204 raft_consensus.cc:2771] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:44.262116 2241 raft_consensus.cc:499] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:44.262312 2241 raft_consensus.cc:521] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:44.263151 2241 leader_election.cc:290] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:44.263796 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "fc173583a9584741b28f2c0203504a3a" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
W20261118 08:54:44.264132 1147 leader_election.cc:343] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a
I20261118 08:54:44.264405 1146 leader_election.cc:304] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:44.264662 2241 raft_consensus.cc:2771] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:44.395051 2241 raft_consensus.cc:499] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:44.395237 2241 raft_consensus.cc:521] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:44.395877 2241 leader_election.cc:290] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:44.396471 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "3e292319f0c44e34b49ae6d579df2ecb" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
W20261118 08:54:44.396667 1147 leader_election.cc:343] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb
I20261118 08:54:44.396729 1075 raft_consensus.cc:2490] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:44.397127 1146 leader_election.cc:304] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters: ad2789ff5b54489c84966cff78be4721
I20261118 08:54:44.397373 2241 raft_consensus.cc:2826] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261118 08:54:44.397498 2241 raft_consensus.cc:499] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:44.397601 2241 raft_consensus.cc:3096] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Advancing to term 2
I20261118 08:54:44.399966 2241 raft_consensus.cc:521] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:44.400686 2241 leader_election.cc:290] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:44.401234 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "3e292319f0c44e34b49ae6d579df2ecb" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:44.401465 1075 raft_consensus.cc:3096] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Advancing to term 2
W20261118 08:54:44.402962 1147 leader_election.cc:343] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb
I20261118 08:54:44.404084 1075 raft_consensus.cc:2490] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 2.
I20261118 08:54:44.404493 1146 leader_election.cc:304] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters: ad2789ff5b54489c84966cff78be4721
I20261118 08:54:44.404811 2250 raft_consensus.cc:2826] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 2 FOLLOWER]: Leader election won for term 2
I20261118 08:54:44.404959 2250 raft_consensus.cc:705] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [term 2 LEADER]: Becoming Leader. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Running, Role: LEADER
I20261118 08:54:44.405443 2250 consensus_queue.cc:238] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } }
I20261118 08:54:44.408420 1075 raft_consensus.cc:1285] T 3e292319f0c44e34b49ae6d579df2ecb P 213b5f7a64e74b109a3a9387e5975042 [term 2 FOLLOWER]: Refusing update from remote peer 3934fbc351ee4a99add35742e90dfa68: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20261118 08:54:44.408514 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:44.409495 2242 consensus_queue.cc:1049] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261118 08:54:44.423717 2204 raft_consensus.cc:499] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:44.423907 2204 raft_consensus.cc:521] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:44.424486 2204 leader_election.cc:290] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:44.424980 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "fc173583a9584741b28f2c0203504a3a" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:44.425171 1211 raft_consensus.cc:2490] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 1.
W20261118 08:54:44.425257 1012 leader_election.cc:343] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a
I20261118 08:54:44.425526 1011 leader_election.cc:304] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters: ad2789ff5b54489c84966cff78be4721
I20261118 08:54:44.425869 2204 raft_consensus.cc:2826] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261118 08:54:44.426002 2204 raft_consensus.cc:499] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:44.426101 2204 raft_consensus.cc:3096] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Advancing to term 2
I20261118 08:54:44.428121 2204 raft_consensus.cc:521] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:44.428637 2204 leader_election.cc:290] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 election: Requested vote from peers 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:44.429155 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "fc173583a9584741b28f2c0203504a3a" candidate_uuid: "213b5f7a64e74b109a3a9387e5975042" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:44.429349 1211 raft_consensus.cc:3096] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Advancing to term 2
W20261118 08:54:44.429451 1012 leader_election.cc:343] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a
I20261118 08:54:44.431300 1211 raft_consensus.cc:2490] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213b5f7a64e74b109a3a9387e5975042 in term 2.
I20261118 08:54:44.431685 1011 leader_election.cc:304] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters: ad2789ff5b54489c84966cff78be4721
I20261118 08:54:44.431959 2204 raft_consensus.cc:2826] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 2 FOLLOWER]: Leader election won for term 2
I20261118 08:54:44.432335 2204 raft_consensus.cc:705] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [term 2 LEADER]: Becoming Leader. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Running, Role: LEADER
I20261118 08:54:44.432679 2204 consensus_queue.cc:238] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:44.434875 1211 raft_consensus.cc:1285] T fc173583a9584741b28f2c0203504a3a P 3934fbc351ee4a99add35742e90dfa68 [term 2 FOLLOWER]: Refusing update from remote peer 213b5f7a64e74b109a3a9387e5975042: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20261118 08:54:44.435186 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:44.435516 2256 consensus_queue.cc:1049] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261118 08:54:44.438470 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--webserver_interface=127.0.72.190
--webserver_port=41231
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20261118 08:54:44.570493 2261 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:44.570724 2261 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:44.571563 2261 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:44.573730 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:44.573909 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:44.574051 1810 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:44.574301 2261 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20261118 08:54:44.775671 2260 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:44.775975 2260 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:44.776087 2260 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:44.785437 2260 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:44.785568 2260 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:44.785653 2260 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:44.785713 2260 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:44.797622 2260 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.72.190
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:44.799216 2260 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:44.801024 2260 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:44.810016 2269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:44.809978 2270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:44.810720 2272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:44.811125 2260 server_base.cc:1061] running on GCE node
I20261118 08:54:44.812016 2260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:44.813534 2260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:44.814772 2260 hybrid_clock.cc:648] HybridClock initialized: now 1794992084814718 us; error 40 us; skew 500 ppm
I20261118 08:54:44.815150 2260 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:44.817447 2260 webserver.cc:577] Webserver started at http://127.0.72.190:41231/ using document root <none> and password file <none>
I20261118 08:54:44.818176 2260 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:44.818279 2260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:44.824108 2260 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20261118 08:54:44.827350 2278 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:44.828881 2260 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20261118 08:54:44.829056 2260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:44.829788 2260 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:44.849324 2260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:44.850241 2260 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:44.850518 2260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:44.869256 2260 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.190:42341
I20261118 08:54:44.869275 2329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.190:42341 every 8 connection(s)
I20261118 08:54:44.870968 2260 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
I20261118 08:54:44.877192 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 2260
I20261118 08:54:44.878539 2330 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:44.882647 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap starting.
I20261118 08:54:44.896541 2330 log.cc:826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:44.938407 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=49 ignored=0} inserts{seen=34 ignored=0} mutations{seen=37 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20261118 08:54:44.938997 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap complete.
I20261118 08:54:44.946107 2330 raft_consensus.cc:360] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:44.948501 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:44.949528 2330 raft_consensus.cc:748] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Initialized, Role: FOLLOWER
I20261118 08:54:44.950356 2330 consensus_queue.cc:261] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 49, Last appended: 5.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:44.951246 2336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:44.951610 2336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:44.952493 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:44.952653 2330 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:44.966886 2350 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:44.967099 2350 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: loading cluster ID for follower catalog manager: success
I20261118 08:54:44.971984 2350 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: acquiring CA information for follower catalog manager: success
I20261118 08:54:44.975409 2350 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:44.996165 2260 master_runner.cc:187] Error getting master registration for 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111),
I20261118 08:54:45.001827 2260 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:45.133347 2250 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:45.133574 2250 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:45.134302 2250 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:54:45.135304 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:54:45.135563 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:54:45.135708 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:45.135991 2250 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:45.138609 984 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:45.140461 2295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:49285; Asking this server to re-register.
I20261118 08:54:45.141134 984 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:45.142781 2295 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:45.171938 2250 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:45.172159 2250 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.172801 2250 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:54:45.173719 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:54:45.173854 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:45.174016 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:45.174291 2250 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:45.266217 1257 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:45.268138 2295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:49553; Asking this server to re-register.
I20261118 08:54:45.268877 1257 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:45.270509 2295 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:45.456005 2295 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:54539; Asking this server to re-register.
I20261118 08:54:45.456629 1122 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:45.457814 2295 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:45.562240 2357 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:45.562467 2357 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:45.564275 2357 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:45.564788 1835 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
I20261118 08:54:45.564999 1835 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 5.
I20261118 08:54:45.565582 2280 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters:
W20261118 08:54:45.565613 2279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:45.565954 2357 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20261118 08:54:45.566066 2357 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:45.566169 2357 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 5 FOLLOWER]: Advancing to term 6
W20261118 08:54:45.567098 2279 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:45.569499 2357 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:45.570129 2357 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 6 election: Requested vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:45.570554 1835 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca"
I20261118 08:54:45.570758 1835 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 5 FOLLOWER]: Advancing to term 6
W20261118 08:54:45.571403 2279 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:45.572911 1835 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 6.
I20261118 08:54:45.573312 2280 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1
I20261118 08:54:45.573601 2357 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Leader election won for term 6
I20261118 08:54:45.573964 2357 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 LEADER]: Becoming Leader. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Running, Role: LEADER
I20261118 08:54:45.574389 2357 consensus_queue.cc:238] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 5.50, Last appended by leader: 50, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:45.576934 2360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 6 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:45.577244 2360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:45.579406 1835 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Refusing update from remote peer 78d4026bc230421ea10b8c68a9ef7871: Log matching property violated. Preceding OpId in replica: term: 5 index: 50. Preceding OpId from leader: term: 6 index: 51. (index mismatch)
W20261118 08:54:45.579671 2279 consensus_peers.cc:597] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 -> Peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Couldn't send request to peer 6c7d9f52503f41e4bafe2821e6fd6af1. Status: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:45.580118 2357 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20261118 08:54:45.581964 2366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 6 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:45.582159 2366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:45.585362 2366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:45.585556 2366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:45.586051 2357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:45.586223 2357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:45.586616 2364 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 08:54:45.589028 2364 catalog_manager.cc:694] Loaded metadata for table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]
I20261118 08:54:45.589435 2364 catalog_manager.cc:694] Loaded metadata for table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]
I20261118 08:54:45.589745 2364 catalog_manager.cc:694] Loaded metadata for table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]
I20261118 08:54:45.590025 2364 catalog_manager.cc:694] Loaded metadata for table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]
I20261118 08:54:45.590315 2364 catalog_manager.cc:694] Loaded metadata for table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]
I20261118 08:54:45.590612 2364 catalog_manager.cc:694] Loaded metadata for table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]
I20261118 08:54:45.590884 2364 catalog_manager.cc:694] Loaded metadata for table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]
I20261118 08:54:45.591161 2364 catalog_manager.cc:694] Loaded metadata for table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]
I20261118 08:54:45.591490 2364 catalog_manager.cc:694] Loaded metadata for table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]
I20261118 08:54:45.593555 2364 tablet_loader.cc:96] loaded metadata for tablet 368ffd5a7d724f47abeea8f223415eac (table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a])
I20261118 08:54:45.594010 2364 tablet_loader.cc:96] loaded metadata for tablet 3c8c4245c7ac4d8ca2e7fa521b6d49d3 (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2])
I20261118 08:54:45.594566 2364 tablet_loader.cc:96] loaded metadata for tablet 50744c2d59124a22ac0a76f2f8e60974 (table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30])
I20261118 08:54:45.594828 2364 tablet_loader.cc:96] loaded metadata for tablet 543333f1b0df4517a424aa536902aa29 (table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a])
I20261118 08:54:45.595269 2364 tablet_loader.cc:96] loaded metadata for tablet 748947f292334e7a97abfa599634006d (table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b])
I20261118 08:54:45.595700 2364 tablet_loader.cc:96] loaded metadata for tablet 86bf01af75674f5491b024ac43a063b0 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957])
I20261118 08:54:45.596383 2364 tablet_loader.cc:96] loaded metadata for tablet c02766f2f8d94f479e0ce2dc03567a7d (table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8])
I20261118 08:54:45.596617 2364 tablet_loader.cc:96] loaded metadata for tablet d2ce64bc39634b5882b1beaa68e66cbe (table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7])
I20261118 08:54:45.596876 2364 tablet_loader.cc:96] loaded metadata for tablet d546b7f20a7f4d3b9e5612080df04be6 (table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70])
I20261118 08:54:45.597560 2364 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 08:54:45.598541 2364 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:45.598621 2364 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 08:54:45.599884 2364 catalog_manager.cc:1555] Loading token signing keys...
I20261118 08:54:45.600752 2364 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Loaded TSK: 0
I20261118 08:54:45.601557 2364 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 08:54:45.602424 2364 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 08:54:45.603241 2364 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 08:54:45.647493 2294 master_service.cc:965] ReplaceTablet: received request to replace tablet 3e292319f0c44e34b49ae6d579df2ecb from {username='slave'} at 127.0.0.1:44548
I20261118 08:54:45.647660 2293 master_service.cc:965] ReplaceTablet: received request to replace tablet c02766f2f8d94f479e0ce2dc03567a7d from {username='slave'} at 127.0.0.1:44548
I20261118 08:54:45.647539 2295 master_service.cc:965] ReplaceTablet: received request to replace tablet 543333f1b0df4517a424aa536902aa29 from {username='slave'} at 127.0.0.1:44548
I20261118 08:54:45.648473 2292 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_567d34233a4b440a952301f19a6bcb10" } modify_external_catalogs: true
I20261118 08:54:45.648810 2292 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_567d34233a4b440a952301f19a6bcb10" } modify_external_catalogs: true
I20261118 08:54:45.648780 2291 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44548:
name: "default.Table_bfd124a61a904e60bc99bee8716f64cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:45.649781 2294 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44548:
name: "default.Table_52bba9536ae942a59235d8087d61576b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:45.650588 2290 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44548:
name: "default.Table_903cefd8f2a74c89be4bf41b84d26ba4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:45.651674 2288 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_294f03c6875c4da0845888273ea80a2a" } modify_external_catalogs: true
I20261118 08:54:45.651759 2289 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_1ee6cacf31104df284b253928039759b" } new_table_name: "default.Table_29f622723bbb4a47970de8920bf7d580" modify_external_catalogs: true
I20261118 08:54:45.651922 2288 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_294f03c6875c4da0845888273ea80a2a" } modify_external_catalogs: true
I20261118 08:54:45.652081 2287 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44548:
name: "default.Table_b5863ec6b7e144f58594f23a0634bf0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:45.652942 2286 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
I20261118 08:54:45.653156 2286 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_c3f2df1b938946c19c1ca00d8fcbd640" } modify_external_catalogs: true
F20261118 08:54:45.657433 2349 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_bc1f66261ac64a87912e19a32ce0ca44, message={"server":"","servicePrincipal":"","db":"default","table":"table_bc1f66261ac64a87912e19a32ce0ca44","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_bc1f66261ac64a87912e19a32ce0ca44\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_bc1f66261ac64a87912e19a32ce0ca44\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_bc1f66261ac64a87912e19a32ce0ca44\",\"kudu.table_id\":\"4fcf984156584e30aac26283f71adda2\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_8b08d3e4ed824f1680377a6d5e7518a8, message={"server":"","servicePrincipal":"","db":"default","table":"table_8b08d3e4ed824f1680377a6d5e7518a8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_8b08d3e4ed824f1680377a6d5e7518a8\",\"kudu.table_id\":\"b42eb739941947f8b7d79fddd923eb52\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20261118 08:54:45.660223 2295 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 543333f1b0df4517a424aa536902aa29
@ 0x7f0dbe599d5d google::LogMessage::Fail() at ??:0
I20261118 08:54:45.661126 2293 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet c02766f2f8d94f479e0ce2dc03567a7d
I20261118 08:54:45.661912 2293 catalog_manager.cc:6936] ReplaceTablet: tablet c02766f2f8d94f479e0ce2dc03567a7d deleted and replaced by tablet 4eecc49bfe014d1d86c84e8a4f16a5bf
I20261118 08:54:45.661980 2295 catalog_manager.cc:6936] ReplaceTablet: tablet 543333f1b0df4517a424aa536902aa29 deleted and replaced by tablet 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:54:45.662235 2295 master_service.cc:965] ReplaceTablet: received request to replace tablet fc173583a9584741b28f2c0203504a3a from {username='slave'} at 127.0.0.1:44548
@ 0x7f0dbe59db23 google::LogMessage::SendToLog() at ??:0
I20261118 08:54:45.662334 2293 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44548:
table { table_name: "default.Table_ef26d2c593994e118ff3d01a0f5a9a1d" } new_table_name: "default.Table_0d932fab32a1414582d1f8370b6e2b23" modify_external_catalogs: true
I20261118 08:54:45.662700 2295 master_service.cc:965] ReplaceTablet: received request to replace tablet 3e292319f0c44e34b49ae6d579df2ecb from {username='slave'} at 127.0.0.1:44548
I20261118 08:54:45.663203 2295 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44548:
name: "default.Table_738b3d8aface40768cac1e7b01d47046"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f0dbe59975c google::LogMessage::Flush() at ??:0
@ 0x7f0dbe5999a1 google::LogMessage::~LogMessage() at ??:0
I20261118 08:54:45.679201 1191 tablet_service.cc:1547] Processing DeleteTablet for tablet c02766f2f8d94f479e0ce2dc03567a7d with delete_type TABLET_DATA_DELETED (replaced by tablet 4eecc49bfe014d1d86c84e8a4f16a5bf) from {username='slave'} at 127.0.0.1:55098
I20261118 08:54:45.679340 1189 tablet_service.cc:1547] Processing DeleteTablet for tablet 543333f1b0df4517a424aa536902aa29 with delete_type TABLET_DATA_DELETED (replaced by tablet 48a6e2d96fe442b6a84b4773993b4f54) from {username='slave'} at 127.0.0.1:55098
I20261118 08:54:45.684916 2379 tablet_replica.cc:333] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:45.685117 2379 raft_consensus.cc:2261] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:45.685356 2379 raft_consensus.cc:2290] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:45.685927 2379 ts_tablet_manager.cc:1962] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:45.687873 1189 tablet_service.cc:1500] Processing CreateTablet for tablet d2ce64bc39634b5882b1beaa68e66cbe (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.688336 1189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ce64bc39634b5882b1beaa68e66cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:45.692135 1188 tablet_service.cc:1500] Processing CreateTablet for tablet 48a6e2d96fe442b6a84b4773993b4f54 (DEFAULT_TABLE table=default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.692721 1188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a6e2d96fe442b6a84b4773993b4f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:45.696115 915 tablet_service.cc:1547] Processing DeleteTablet for tablet c02766f2f8d94f479e0ce2dc03567a7d with delete_type TABLET_DATA_DELETED (replaced by tablet 4eecc49bfe014d1d86c84e8a4f16a5bf) from {username='slave'} at 127.0.0.1:37264
I20261118 08:54:45.696417 917 tablet_service.cc:1547] Processing DeleteTablet for tablet 543333f1b0df4517a424aa536902aa29 with delete_type TABLET_DATA_DELETED (replaced by tablet 48a6e2d96fe442b6a84b4773993b4f54) from {username='slave'} at 127.0.0.1:37264
I20261118 08:54:45.696815 914 tablet_service.cc:1500] Processing CreateTablet for tablet 48a6e2d96fe442b6a84b4773993b4f54 (DEFAULT_TABLE table=default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.696942 918 tablet_service.cc:1500] Processing CreateTablet for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.697422 914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a6e2d96fe442b6a84b4773993b4f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:45.697602 916 tablet_service.cc:1500] Processing CreateTablet for tablet d2ce64bc39634b5882b1beaa68e66cbe (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.697970 916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ce64bc39634b5882b1beaa68e66cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:45.702206 2379 ts_tablet_manager.cc:1975] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:45.702330 2379 log.cc:1199] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/c02766f2f8d94f479e0ce2dc03567a7d
I20261118 08:54:45.702744 2379 ts_tablet_manager.cc:1996] T c02766f2f8d94f479e0ce2dc03567a7d P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:45.703578 918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:45.703809 2379 tablet_replica.cc:333] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: stopping tablet replica
I20261118 08:54:45.704002 2379 raft_consensus.cc:2261] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:45.704208 2379 raft_consensus.cc:2290] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:45.704780 2379 ts_tablet_manager.cc:1962] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:45.709052 2281 catalog_manager.cc:5065] TS 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): tablet c02766f2f8d94f479e0ce2dc03567a7d (table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]) successfully deleted
I20261118 08:54:45.710289 1054 tablet_service.cc:1547] Processing DeleteTablet for tablet 543333f1b0df4517a424aa536902aa29 with delete_type TABLET_DATA_DELETED (replaced by tablet 48a6e2d96fe442b6a84b4773993b4f54) from {username='slave'} at 127.0.0.1:42714
I20261118 08:54:45.711205 1055 tablet_service.cc:1547] Processing DeleteTablet for tablet c02766f2f8d94f479e0ce2dc03567a7d with delete_type TABLET_DATA_DELETED (replaced by tablet 4eecc49bfe014d1d86c84e8a4f16a5bf) from {username='slave'} at 127.0.0.1:42714
I20261118 08:54:45.711453 1053 tablet_service.cc:1500] Processing CreateTablet for tablet d2ce64bc39634b5882b1beaa68e66cbe (DEFAULT_TABLE table=default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.711581 1056 tablet_service.cc:1500] Processing CreateTablet for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.711984 1053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2ce64bc39634b5882b1beaa68e66cbe. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f0dd3a199a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 08:54:45.712085 1052 tablet_service.cc:1500] Processing CreateTablet for tablet 48a6e2d96fe442b6a84b4773993b4f54 (DEFAULT_TABLE table=default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.712599 1052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a6e2d96fe442b6a84b4773993b4f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:45.715615 1056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f0dd3a18251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20261118 08:54:45.717371 2379 ts_tablet_manager.cc:1975] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:45.717506 2379 log.cc:1199] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-2/wal/wals/543333f1b0df4517a424aa536902aa29
I20261118 08:54:45.717850 2379 ts_tablet_manager.cc:1996] T 543333f1b0df4517a424aa536902aa29 P 3934fbc351ee4a99add35742e90dfa68: Deleting consensus metadata
I20261118 08:54:45.719834 2281 catalog_manager.cc:5065] TS 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): tablet 543333f1b0df4517a424aa536902aa29 (table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]) successfully deleted
I20261118 08:54:45.723469 2384 tablet_bootstrap.cc:492] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:45.725718 2384 tablet_bootstrap.cc:654] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
@ 0x7f0dd3a1fecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7f0dd3a1fd49 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 08:54:45.735297 1191 tablet_service.cc:1500] Processing CreateTablet for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:45.735878 1191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f0dbf9e2dc9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0dc30bd6db start_thread at ??:0
@ 0x7f0dbbb2d61f clone at ??:0
I20261118 08:54:45.768440 2387 tablet_bootstrap.cc:492] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:45.770736 2387 tablet_bootstrap.cc:654] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.771905 2383 tablet_replica.cc:333] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:45.772131 2383 raft_consensus.cc:2261] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:45.772547 2383 raft_consensus.cc:2290] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:45.773263 2383 ts_tablet_manager.cc:1962] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:45.777240 2385 tablet_bootstrap.cc:492] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:45.779354 2385 tablet_bootstrap.cc:654] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.780836 2382 tablet_replica.cc:333] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:45.781045 2382 raft_consensus.cc:2261] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:45.781320 2382 raft_consensus.cc:2290] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:45.781953 2382 ts_tablet_manager.cc:1962] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:45.782316 2387 tablet_bootstrap.cc:492] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:45.782378 2384 tablet_bootstrap.cc:492] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:45.782455 2387 ts_tablet_manager.cc:1449] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20261118 08:54:45.782536 2384 ts_tablet_manager.cc:1449] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.019s
I20261118 08:54:45.783213 2387 raft_consensus.cc:360] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.783420 2384 raft_consensus.cc:360] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.783624 2384 raft_consensus.cc:386] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.783722 2384 raft_consensus.cc:748] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.783979 2384 consensus_queue.cc:261] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
W20261118 08:54:45.785959 1011 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.72.131:46447, user_credentials={real_user=slave}} blocked reactor thread for 102812us
I20261118 08:54:45.786330 2387 raft_consensus.cc:386] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.786446 2387 raft_consensus.cc:748] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.786679 2387 consensus_queue.cc:261] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
W20261118 08:54:45.787444 1011 consensus_peers.cc:597] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 -> Peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Couldn't send request to peer 3934fbc351ee4a99add35742e90dfa68. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c02766f2f8d94f479e0ce2dc03567a7d. This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:45.788435 2382 ts_tablet_manager.cc:1975] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:45.788546 2382 log.cc:1199] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/c02766f2f8d94f479e0ce2dc03567a7d
I20261118 08:54:45.788952 2382 ts_tablet_manager.cc:1996] T c02766f2f8d94f479e0ce2dc03567a7d P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:45.790069 2383 ts_tablet_manager.cc:1975] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:45.790195 2383 log.cc:1199] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/543333f1b0df4517a424aa536902aa29
I20261118 08:54:45.790598 2383 ts_tablet_manager.cc:1996] T 543333f1b0df4517a424aa536902aa29 P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:45.790830 2385 tablet_bootstrap.cc:492] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:45.790961 2385 ts_tablet_manager.cc:1449] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20261118 08:54:45.791806 2385 raft_consensus.cc:360] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.791985 2385 raft_consensus.cc:386] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.792059 2385 raft_consensus.cc:748] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.792346 2385 consensus_queue.cc:261] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.792900 2385 ts_tablet_manager.cc:1480] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261118 08:54:45.793260 2387 ts_tablet_manager.cc:1480] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20261118 08:54:45.793540 2387 tablet_bootstrap.cc:492] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:45.795266 2385 tablet_bootstrap.cc:492] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:45.795974 2387 tablet_bootstrap.cc:654] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.796962 2385 tablet_bootstrap.cc:654] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.797372 984 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:45.797889 2382 tablet_replica.cc:333] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:45.798087 2382 raft_consensus.cc:2261] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:45.798280 2384 ts_tablet_manager.cc:1480] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20261118 08:54:45.798300 2382 raft_consensus.cc:2290] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:45.798563 2384 tablet_bootstrap.cc:492] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:45.798717 2383 tablet_replica.cc:333] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:45.798894 2383 raft_consensus.cc:2261] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:45.798913 2382 ts_tablet_manager.cc:1962] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:45.799203 2383 raft_consensus.cc:2290] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:45.799755 2383 ts_tablet_manager.cc:1962] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:45.800385 2384 tablet_bootstrap.cc:654] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.807165 2387 tablet_bootstrap.cc:492] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:45.807340 2387 ts_tablet_manager.cc:1449] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20261118 08:54:45.808202 2387 raft_consensus.cc:360] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.808408 2387 raft_consensus.cc:386] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.808467 2387 raft_consensus.cc:748] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.808926 2387 consensus_queue.cc:261] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.809528 2387 ts_tablet_manager.cc:1480] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 08:54:45.809882 2387 tablet_bootstrap.cc:492] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68: Bootstrap starting.
I20261118 08:54:45.810953 2382 ts_tablet_manager.cc:1975] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:45.811091 2382 log.cc:1199] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/543333f1b0df4517a424aa536902aa29
I20261118 08:54:45.811533 2382 ts_tablet_manager.cc:1996] T 543333f1b0df4517a424aa536902aa29 P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:45.811796 2387 tablet_bootstrap.cc:654] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.813758 2385 tablet_bootstrap.cc:492] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:45.813923 2385 ts_tablet_manager.cc:1449] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.001s
I20261118 08:54:45.814698 2385 raft_consensus.cc:360] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.814860 2383 ts_tablet_manager.cc:1975] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:45.814904 2385 raft_consensus.cc:386] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.814980 2385 raft_consensus.cc:748] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.814983 2383 log.cc:1199] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/c02766f2f8d94f479e0ce2dc03567a7d
I20261118 08:54:45.815599 2383 ts_tablet_manager.cc:1996] T c02766f2f8d94f479e0ce2dc03567a7d P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:45.816249 2384 tablet_bootstrap.cc:492] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:45.816320 2387 tablet_bootstrap.cc:492] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68: No bootstrap required, opened a new log
I20261118 08:54:45.816408 2384 ts_tablet_manager.cc:1449] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20261118 08:54:45.816449 2387 ts_tablet_manager.cc:1449] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20261118 08:54:45.816603 2385 consensus_queue.cc:261] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.817167 2384 raft_consensus.cc:360] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.817354 2384 raft_consensus.cc:386] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.817420 2384 raft_consensus.cc:748] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.817492 2385 ts_tablet_manager.cc:1480] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20261118 08:54:45.817646 2384 consensus_queue.cc:261] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.817993 2385 tablet_bootstrap.cc:492] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:45.817994 2387 raft_consensus.cc:360] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.818112 2384 ts_tablet_manager.cc:1480] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 08:54:45.818153 2387 raft_consensus.cc:386] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.818243 2387 raft_consensus.cc:748] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.818406 2384 tablet_bootstrap.cc:492] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: Bootstrap starting.
I20261118 08:54:45.818445 2387 consensus_queue.cc:261] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.818929 2387 ts_tablet_manager.cc:1480] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261118 08:54:45.819768 2385 tablet_bootstrap.cc:654] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.820111 2384 tablet_bootstrap.cc:654] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:45.824848 2384 tablet_bootstrap.cc:492] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: No bootstrap required, opened a new log
I20261118 08:54:45.824980 2384 ts_tablet_manager.cc:1449] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20261118 08:54:45.825731 2384 raft_consensus.cc:360] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.825886 2384 raft_consensus.cc:386] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.825963 2384 raft_consensus.cc:748] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.826211 2384 consensus_queue.cc:261] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.826809 2384 ts_tablet_manager.cc:1480] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20261118 08:54:45.828231 2385 tablet_bootstrap.cc:492] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:45.828397 2385 ts_tablet_manager.cc:1449] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20261118 08:54:45.829315 2385 raft_consensus.cc:360] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.829492 2385 raft_consensus.cc:386] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:45.829578 2385 raft_consensus.cc:748] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:45.829788 2385 consensus_queue.cc:261] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.830346 2385 ts_tablet_manager.cc:1480] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20261118 08:54:45.902840 2241 raft_consensus.cc:499] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:45.903061 2241 raft_consensus.cc:521] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.903960 2241 leader_election.cc:290] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:45.904490 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4eecc49bfe014d1d86c84e8a4f16a5bf" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:45.904538 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4eecc49bfe014d1d86c84e8a4f16a5bf" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:45.904729 1075 raft_consensus.cc:2490] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:45.904806 938 raft_consensus.cc:2490] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:45.905341 1146 leader_election.cc:304] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, 3934fbc351ee4a99add35742e90dfa68; no voters:
I20261118 08:54:45.905652 2241 raft_consensus.cc:2826] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:45.905769 2241 raft_consensus.cc:499] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:45.905879 2241 raft_consensus.cc:3096] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:45.907871 2241 raft_consensus.cc:521] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.908429 2241 leader_election.cc:290] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:45.908981 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4eecc49bfe014d1d86c84e8a4f16a5bf" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:45.909003 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4eecc49bfe014d1d86c84e8a4f16a5bf" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:45.909184 938 raft_consensus.cc:3096] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:45.909202 1075 raft_consensus.cc:3096] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:45.912144 938 raft_consensus.cc:2490] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:45.912132 1075 raft_consensus.cc:2490] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:45.912635 1147 leader_election.cc:304] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:45.912974 2241 raft_consensus.cc:2826] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:45.913124 2241 raft_consensus.cc:705] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [term 1 LEADER]: Becoming Leader. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Running, Role: LEADER
I20261118 08:54:45.913421 2241 consensus_queue.cc:238] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:45.916931 2250 consensus_queue.cc:1049] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:45.917296 2241 consensus_queue.cc:1049] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261118 08:54:45.969353 1147 connection.cc:573] server connection from 127.0.0.1:55098 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:45.969331 871 connection.cc:573] server connection from 127.0.0.1:37264 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:45.969583 1145 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:45.969689 872 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:45.969868 1010 connection.cc:573] server connection from 127.0.0.1:42714 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:45.971330 1010 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:45.971489 2240 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:45.970600 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.190:42341 (local address 127.0.72.131:60079)
1118 08:54:45.970873 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:45.970893 (+ 20us) client_negotiation.cc:175] Beginning negotiation
1118 08:54:45.971022 (+ 129us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 08:54:45.971440 (+ 418us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":158}
W20261118 08:54:45.971405 1264 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:45.972258 1264 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.72.190:42341)
W20261118 08:54:45.972851 1264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:45.973102 1264 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:45.973062 2226 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:45.970722 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.72.190:42341 (local address 127.0.72.129:36069)
1118 08:54:45.971022 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1118 08:54:45.971036 (+ 14us) client_negotiation.cc:175] Beginning negotiation
1118 08:54:45.971128 (+ 92us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
1118 08:54:45.973015 (+ 1887us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":197}
W20261118 08:54:45.973461 1264 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.72.190:42341)
I20261118 08:54:46.153340 2391 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:46.153543 2391 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:46.154693 2391 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:46.155129 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "48a6e2d96fe442b6a84b4773993b4f54" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:46.155332 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "48a6e2d96fe442b6a84b4773993b4f54" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68" is_pre_election: true
I20261118 08:54:46.155421 1076 raft_consensus.cc:2490] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:46.155586 1211 raft_consensus.cc:2490] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 0.
I20261118 08:54:46.155867 873 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:46.156147 2391 raft_consensus.cc:2826] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:46.156244 2391 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:46.156328 2391 raft_consensus.cc:3096] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:46.158239 2391 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:46.158764 2391 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Requested vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:46.159190 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "48a6e2d96fe442b6a84b4773993b4f54" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:46.159279 1211 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "48a6e2d96fe442b6a84b4773993b4f54" candidate_uuid: "ad2789ff5b54489c84966cff78be4721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934fbc351ee4a99add35742e90dfa68"
I20261118 08:54:46.159404 1076 raft_consensus.cc:3096] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:46.159466 1211 raft_consensus.cc:3096] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:46.161302 1076 raft_consensus.cc:2490] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:46.161321 1211 raft_consensus.cc:2490] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2789ff5b54489c84966cff78be4721 in term 1.
I20261118 08:54:46.161687 873 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:46.161967 2391 raft_consensus.cc:2826] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:46.162343 2391 raft_consensus.cc:705] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Becoming Leader. State: Replica: ad2789ff5b54489c84966cff78be4721, State: Running, Role: LEADER
I20261118 08:54:46.162667 2391 consensus_queue.cc:238] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:46.165011 2241 raft_consensus.cc:499] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:46.165171 2241 raft_consensus.cc:521] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:46.165762 2241 leader_election.cc:290] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:46.166210 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d2ce64bc39634b5882b1beaa68e66cbe" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721" is_pre_election: true
I20261118 08:54:46.166421 938 raft_consensus.cc:2490] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:46.166862 1147 leader_election.cc:304] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:46.166977 2413 consensus_queue.cc:1049] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:46.166971 1076 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d2ce64bc39634b5882b1beaa68e66cbe" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042" is_pre_election: true
I20261118 08:54:46.167188 1076 raft_consensus.cc:2490] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 0.
I20261118 08:54:46.167236 2241 raft_consensus.cc:2826] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261118 08:54:46.167337 2241 raft_consensus.cc:499] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:46.167464 2241 raft_consensus.cc:3096] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:46.167888 2413 consensus_queue.cc:1049] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:46.170892 2241 raft_consensus.cc:521] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:46.171948 938 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d2ce64bc39634b5882b1beaa68e66cbe" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2789ff5b54489c84966cff78be4721"
I20261118 08:54:46.172158 938 raft_consensus.cc:3096] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:46.174952 938 raft_consensus.cc:2490] T d2ce64bc39634b5882b1beaa68e66cbe P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:46.175511 1147 leader_election.cc:304] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721; no voters:
I20261118 08:54:46.175818 2250 raft_consensus.cc:2826] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader election won for term 1
I20261118 08:54:46.175943 2250 raft_consensus.cc:705] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [term 1 LEADER]: Becoming Leader. State: Replica: 3934fbc351ee4a99add35742e90dfa68, State: Running, Role: LEADER
I20261118 08:54:46.175967 1075 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "d2ce64bc39634b5882b1beaa68e66cbe" candidate_uuid: "3934fbc351ee4a99add35742e90dfa68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213b5f7a64e74b109a3a9387e5975042"
I20261118 08:54:46.176162 1075 raft_consensus.cc:3096] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Advancing to term 1
I20261118 08:54:46.176242 2250 consensus_queue.cc:238] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:46.180442 1075 raft_consensus.cc:2490] T d2ce64bc39634b5882b1beaa68e66cbe P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3934fbc351ee4a99add35742e90dfa68 in term 1.
I20261118 08:54:46.180924 2241 leader_election.cc:290] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 1 election: Requested vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:46.194816 2421 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:46.195036 2421 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:46.197327 2241 consensus_queue.cc:1049] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:46.197964 2241 consensus_queue.cc:1049] T d2ce64bc39634b5882b1beaa68e66cbe P 3934fbc351ee4a99add35742e90dfa68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261118 08:54:46.204421 2421 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:46.206439 1809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20261118 08:54:46.223146 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:46.224149 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:46.224355 1810 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:46.224874 2421 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 08:54:46.882912 2436 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:46.883109 2436 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:46.885046 2436 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:46.887039 1809 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:46.887231 1810 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:46.887413 1810 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:46.887705 2436 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 08:54:47.057238 290 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task3eSafg/build/asan/bin/kudu pid 1790
I20261118 08:54:47.063370 1855 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 08:54:47.177846 290 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task3eSafg/build/asan/bin/kudu with pid 1790
I20261118 08:54:48.164377 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--webserver_interface=127.0.72.189
--webserver_port=35929
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 08:54:48.501622 2458 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:48.501938 2458 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:48.502033 2458 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:48.510529 2458 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:48.510651 2458 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:48.510743 2458 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:48.510811 2458 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:48.520977 2458 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.72.189
--webserver_port=35929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:48.522761 2458 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:48.524560 2458 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:48.533233 2469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:48.533429 2470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:48.533483 2472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:48.534878 2458 server_base.cc:1061] running on GCE node
I20261118 08:54:48.535692 2458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:48.537107 2458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:48.538306 2458 hybrid_clock.cc:648] HybridClock initialized: now 1794992088538248 us; error 50 us; skew 500 ppm
I20261118 08:54:48.538630 2458 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:48.540664 2458 webserver.cc:577] Webserver started at http://127.0.72.189:35929/ using document root <none> and password file <none>
I20261118 08:54:48.541405 2458 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:48.541508 2458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:48.546828 2458 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20261118 08:54:48.549556 2478 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:48.550855 2458 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20261118 08:54:48.551024 2458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
uuid: "857a577b88ba46cc846ecd1a25a722ca"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:48.551762 2458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:48.598577 2458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:48.599639 2458 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:48.599933 2458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:48.620504 2458 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.189:33939
I20261118 08:54:48.620548 2529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.189:33939 every 8 connection(s)
I20261118 08:54:48.622169 2458 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
I20261118 08:54:48.622855 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 2458
I20261118 08:54:48.630662 2530 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:48.635027 2530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap starting.
W20261118 08:54:48.641742 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 6: this message will repeat every 5th retry.
I20261118 08:54:48.647533 1258 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:48.652526 2530 log.cc:826] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:48.700654 2530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=36 ignored=0} mutations{seen=45 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 08:54:48.701287 2530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap complete.
I20261118 08:54:48.706863 2530 raft_consensus.cc:360] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:48.707620 2530 raft_consensus.cc:748] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857a577b88ba46cc846ecd1a25a722ca, State: Initialized, Role: FOLLOWER
I20261118 08:54:48.708285 2530 consensus_queue.cc:261] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 6.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:48.709052 2534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:48.709292 2534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:48.710232 2530 sys_catalog.cc:565] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:48.722970 2548 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:48.723174 2548 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: loading cluster ID for follower catalog manager: success
I20261118 08:54:48.728024 2548 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: acquiring CA information for follower catalog manager: success
I20261118 08:54:48.731308 2548 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:48.750785 2495 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:41845; Asking this server to re-register.
I20261118 08:54:48.751885 1258 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:48.736522 2458 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:54:48.752414 2458 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:48.752772 985 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:48.754333 2495 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:48.755697 2494 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:48487; Asking this server to re-register.
I20261118 08:54:48.756332 985 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:48.757711 2494 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:48.806824 1123 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:48.808385 2494 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:48955; Asking this server to re-register.
I20261118 08:54:48.809010 1123 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:48.810353 2494 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:54:49.143632 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 6: this message will repeat every 5th retry.
W20261118 08:54:49.320322 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 6: this message will repeat every 5th retry.
I20261118 08:54:49.321818 2561 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:49.322079 2561 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:49.323889 2561 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:49.324710 2480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:49.325904 2479 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:49.326093 2480 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:49.326259 2480 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:49.326573 2561 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20261118 08:54:49.810585 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 6: this message will repeat every 5th retry.
I20261118 08:54:49.933621 2567 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:49.933812 2567 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:49.934547 2567 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:49.938050 2480 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:49.938326 2479 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:49.938469 2479 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:49.938804 2567 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 08:54:50.522322 2581 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:50.522505 2581 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:50.523126 2581 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:50.526448 2480 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:50.526791 2479 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:50.526935 2479 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:50.527194 2581 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 08:54:50.824267 2242 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:50.824481 2242 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:50.825088 2242 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:54:50.826112 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:54:50.826280 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:50.826400 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:50.826656 2242 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:51.078720 290 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-task3eSafg/build/asan/bin/kudu pid 2458
I20261118 08:54:51.086760 2525 generic_service.cc:196] Checking for leaks (request via RPC)
I20261118 08:54:51.192257 2598 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:51.192257 2593 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:51.192411 290 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task3eSafg/build/asan/bin/kudu with pid 2458
I20261118 08:54:51.192440 2593 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:54:51.193184 2593 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:54:51.194191 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:54:51.194352 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:54:51.194466 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:51.194710 2593 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:54:51.210144 1010 connection.cc:573] client connection to 127.0.72.189:33939 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:51.210884 1264 connection.cc:573] client connection to 127.0.72.189:33939 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
I20261118 08:54:52.007283 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--webserver_interface=127.0.72.189
--webserver_port=35929
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 08:54:52.356608 2617 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:52.356947 2617 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:52.357081 2617 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:52.365967 2617 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:52.366067 2617 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:52.366137 2617 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:52.366214 2617 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:52.377364 2617 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.189:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.72.189
--webserver_port=35929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:52.379067 2617 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:52.380987 2617 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:52.389887 2629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:52.389856 2632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:52.389817 2630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:52.392782 2617 server_base.cc:1061] running on GCE node
I20261118 08:54:52.393589 2617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:52.395141 2617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:52.396345 2617 hybrid_clock.cc:648] HybridClock initialized: now 1794992092396276 us; error 69 us; skew 500 ppm
I20261118 08:54:52.396692 2617 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:52.398834 2617 webserver.cc:577] Webserver started at http://127.0.72.189:35929/ using document root <none> and password file <none>
I20261118 08:54:52.399610 2617 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:52.399724 2617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:52.405570 2617 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20261118 08:54:52.408568 2638 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:52.409999 2617 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261118 08:54:52.410172 2617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
uuid: "857a577b88ba46cc846ecd1a25a722ca"
format_stamp: "Formatted at 2026-11-18 08:54:22 on dist-test-slave-m8c8"
I20261118 08:54:52.410979 2617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:52.423744 2617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:52.424645 2617 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:52.424940 2617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:52.445588 2617 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.189:33939
I20261118 08:54:52.445605 2690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.189:33939 every 8 connection(s)
I20261118 08:54:52.447413 2617 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-1/data/info.pb
I20261118 08:54:52.454308 2691 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:52.456516 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 2617
I20261118 08:54:52.458003 2691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap starting.
I20261118 08:54:52.472829 2691 log.cc:826] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:52.515352 2691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=36 ignored=0} mutations{seen=45 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 08:54:52.515925 2691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: Bootstrap complete.
I20261118 08:54:52.521214 2691 raft_consensus.cc:360] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:52.522043 2691 raft_consensus.cc:748] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857a577b88ba46cc846ecd1a25a722ca, State: Initialized, Role: FOLLOWER
I20261118 08:54:52.522727 2691 consensus_queue.cc:261] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 6.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:52.523522 2695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:52.523782 2695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:52.524540 2691 sys_catalog.cc:565] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:52.537729 2709 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:52.537936 2709 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: loading cluster ID for follower catalog manager: success
I20261118 08:54:52.542081 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:54:52.547641 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:52.547996 2709 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: acquiring CA information for follower catalog manager: success
I20261118 08:54:52.550210 2709 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:52.681993 1258 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:52.685280 2656 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:55959; Asking this server to re-register.
I20261118 08:54:52.686005 1258 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:52.687801 2656 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:52.763193 985 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:52.764703 2656 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:42005; Asking this server to re-register.
I20261118 08:54:52.765357 985 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:52.766754 2656 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:52.862286 1123 heartbeater.cc:345] Connected to a master server at 127.0.72.189:33939
I20261118 08:54:52.863750 2656 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:43047; Asking this server to re-register.
I20261118 08:54:52.864310 1123 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:52.865576 2656 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:53.106848 2718 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:53.107143 2718 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:53.109498 2718 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:53.110328 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:53.111881 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:53.112092 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:53.112221 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:53.112576 2718 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20261118 08:54:53.803686 2732 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:53.803856 2732 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:53.804476 2732 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:53.806480 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:53.806631 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:53.806768 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:53.807036 2732 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20261118 08:54:54.210915 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 6: this message will repeat every 5th retry.
I20261118 08:54:54.676335 2744 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:54.676558 2744 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:54.677285 2744 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:54.679185 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:54.679455 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:54.679593 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:54.679852 2744 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20261118 08:54:55.052481 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 6: this message will repeat every 5th retry.
I20261118 08:54:55.641901 2764 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:55.642112 2764 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:55.642870 2764 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:55.643115 290 external_mini_cluster.cc:1372] Running /tmp/dist-test-task3eSafg/build/asan/bin/kudu
/tmp/dist-test-task3eSafg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--webserver_interface=127.0.72.190
--webserver_port=41231
--builtin_ntp_servers=127.0.72.148:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20261118 08:54:55.645220 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:55.645453 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:55.645566 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:55.645820 2764 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20261118 08:54:55.983181 2768 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261118 08:54:55.983562 2768 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261118 08:54:55.983675 2768 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261118 08:54:55.993059 2768 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261118 08:54:55.993150 2768 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261118 08:54:55.993213 2768 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261118 08:54:55.993279 2768 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261118 08:54:56.003607 2768 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.72.148:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:45085
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.72.190:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.72.190
--webserver_port=41231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type FASTDEBUG
built by None at 18 Nov 2026 08:43:23 UTC on bdcb31816ec0
build id 14841
ASAN enabled
I20261118 08:54:56.005210 2768 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261118 08:54:56.007289 2768 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261118 08:54:56.015755 2779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:56.015780 2780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261118 08:54:56.016764 2782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261118 08:54:56.017274 2768 server_base.cc:1061] running on GCE node
I20261118 08:54:56.017982 2768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261118 08:54:56.019335 2768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261118 08:54:56.020569 2768 hybrid_clock.cc:648] HybridClock initialized: now 1794992096020505 us; error 52 us; skew 500 ppm
I20261118 08:54:56.020874 2768 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20261118 08:54:56.022970 2768 webserver.cc:577] Webserver started at http://127.0.72.190:41231/ using document root <none> and password file <none>
I20261118 08:54:56.023710 2768 fs_manager.cc:362] Metadata directory not provided
I20261118 08:54:56.023805 2768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261118 08:54:56.029352 2768 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20261118 08:54:56.032338 2788 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261118 08:54:56.033831 2768 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261118 08:54:56.034003 2768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data,/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
uuid: "78d4026bc230421ea10b8c68a9ef7871"
format_stamp: "Formatted at 2026-11-18 08:54:21 on dist-test-slave-m8c8"
I20261118 08:54:56.034749 2768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261118 08:54:56.049816 2768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261118 08:54:56.050731 2768 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261118 08:54:56.050995 2768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261118 08:54:56.069643 2768 rpc_server.cc:307] RPC server started. Bound to: 127.0.72.190:42341
I20261118 08:54:56.069659 2839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.72.190:42341 every 8 connection(s)
I20261118 08:54:56.071275 2768 server_base.cc:1193] Dumped server information to /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/master-0/data/info.pb
I20261118 08:54:56.071602 290 external_mini_cluster.cc:1434] Started /tmp/dist-test-task3eSafg/build/asan/bin/kudu as pid 2768
I20261118 08:54:56.079089 2840 sys_catalog.cc:263] Verifying existing consensus state
I20261118 08:54:56.083700 2840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap starting.
I20261118 08:54:56.101110 2840 log.cc:826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Log is configured to *not* fsync() on all Append() calls
I20261118 08:54:56.141664 2840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=36 ignored=0} mutations{seen=45 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261118 08:54:56.142185 2840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Bootstrap complete.
I20261118 08:54:56.149351 2840 raft_consensus.cc:360] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:56.150352 2840 raft_consensus.cc:748] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Initialized, Role: FOLLOWER
I20261118 08:54:56.150998 2840 consensus_queue.cc:261] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 6.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:56.151835 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:56.152213 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:56.153226 2840 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: configured and running, proceeding with master startup.
I20261118 08:54:56.153894 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:56.165248 2861 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:56.165446 2861 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: loading cluster ID for follower catalog manager: success
I20261118 08:54:56.170416 2861 catalog_manager.cc:1625] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: acquiring CA information for follower catalog manager: success
I20261118 08:54:56.173664 2861 catalog_manager.cc:1653] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20261118 08:54:56.201002 2768 master_runner.cc:187] Error getting master registration for 127.0.72.188:41259: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111),
I20261118 08:54:56.204668 2768 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:56.273921 984 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:56.274189 984 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:56.276432 2805 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ad2789ff5b54489c84966cff78be4721" instance_seqno: 1794992064082361) as {username='slave'} at 127.0.72.129:54409; Asking this server to re-register.
I20261118 08:54:56.277091 984 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:56.278643 2805 ts_manager.cc:210] Registered new tserver with Master: ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
I20261118 08:54:56.515724 1257 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:56.517786 2805 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" instance_seqno: 1794992065071421) as {username='slave'} at 127.0.72.131:43045; Asking this server to re-register.
I20261118 08:54:56.518414 1257 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:56.519778 2805 ts_manager.cc:210] Registered new tserver with Master: 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
I20261118 08:54:56.657977 1122 heartbeater.cc:345] Connected to a master server at 127.0.72.190:42341
I20261118 08:54:56.659647 2805 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" instance_seqno: 1794992064582482) as {username='slave'} at 127.0.72.130:36031; Asking this server to re-register.
I20261118 08:54:56.660270 1122 heartbeater.cc:462] Registering TS with master...
I20261118 08:54:56.661495 2805 ts_manager.cc:210] Registered new tserver with Master: 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
I20261118 08:54:56.696574 2874 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:56.696913 2874 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:56.698781 2874 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:56.699431 2789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:56.699493 2666 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 7 candidate_status { last_received { term: 6 index: 54 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca" is_pre_election: true
I20261118 08:54:56.699885 2666 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 6.
I20261118 08:54:56.700531 2790 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters:
I20261118 08:54:56.700898 2874 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20261118 08:54:56.700930 2789 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:56.701074 2874 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261118 08:54:56.701180 2874 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 6 FOLLOWER]: Advancing to term 7
I20261118 08:54:56.704665 2874 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:56.705256 2874 leader_election.cc:290] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 7 election: Requested vote from peers 857a577b88ba46cc846ecd1a25a722ca (127.0.72.189:33939), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
I20261118 08:54:56.705741 2666 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78d4026bc230421ea10b8c68a9ef7871" candidate_term: 7 candidate_status { last_received { term: 6 index: 54 } } ignore_live_leader: false dest_uuid: "857a577b88ba46cc846ecd1a25a722ca"
I20261118 08:54:56.705956 2666 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 6 FOLLOWER]: Advancing to term 7
W20261118 08:54:56.706575 2789 leader_election.cc:336] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:56.709594 2666 raft_consensus.cc:2490] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d4026bc230421ea10b8c68a9ef7871 in term 7.
I20261118 08:54:56.710006 2790 leader_election.cc:304] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78d4026bc230421ea10b8c68a9ef7871, 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1
I20261118 08:54:56.710306 2874 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 7 FOLLOWER]: Leader election won for term 7
I20261118 08:54:56.710738 2874 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [term 7 LEADER]: Becoming Leader. State: Replica: 78d4026bc230421ea10b8c68a9ef7871, State: Running, Role: LEADER
I20261118 08:54:56.711311 2874 consensus_queue.cc:238] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 6.54, Last appended by leader: 54, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:56.714267 2876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 7 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:56.714550 2876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:56.716113 2666 raft_consensus.cc:1285] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Refusing update from remote peer 78d4026bc230421ea10b8c68a9ef7871: Log matching property violated. Preceding OpId in replica: term: 6 index: 54. Preceding OpId from leader: term: 7 index: 55. (index mismatch)
W20261118 08:54:56.716861 2789 consensus_peers.cc:597] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 -> Peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Couldn't send request to peer 6c7d9f52503f41e4bafe2821e6fd6af1. Status: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:56.717177 2878 consensus_queue.cc:1049] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20261118 08:54:56.721215 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d4026bc230421ea10b8c68a9ef7871. Latest consensus state: current_term: 7 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:56.721450 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:56.721443 2880 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261118 08:54:56.722764 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:56.723037 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871 [sys.catalog]: This master's current role is: LEADER
I20261118 08:54:56.724663 2880 catalog_manager.cc:694] Loaded metadata for table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30]
I20261118 08:54:56.725175 2880 catalog_manager.cc:694] Loaded metadata for table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a]
I20261118 08:54:56.725682 2880 catalog_manager.cc:694] Loaded metadata for table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70]
I20261118 08:54:56.726058 2880 catalog_manager.cc:694] Loaded metadata for table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a]
I20261118 08:54:56.726359 2880 catalog_manager.cc:694] Loaded metadata for table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2]
I20261118 08:54:56.726704 2880 catalog_manager.cc:694] Loaded metadata for table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b]
I20261118 08:54:56.726996 2880 catalog_manager.cc:694] Loaded metadata for table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7]
I20261118 08:54:56.727305 2880 catalog_manager.cc:694] Loaded metadata for table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957]
I20261118 08:54:56.727635 2880 catalog_manager.cc:694] Loaded metadata for table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]
I20261118 08:54:56.729702 2880 tablet_loader.cc:96] loaded metadata for tablet 368ffd5a7d724f47abeea8f223415eac (table default.table_1ee6cacf31104df284b253928039759b [id=1950821cc4b84c97939b8da6baaab97a])
I20261118 08:54:56.730134 2880 tablet_loader.cc:96] loaded metadata for tablet 3c8c4245c7ac4d8ca2e7fa521b6d49d3 (table default.table_bc1f66261ac64a87912e19a32ce0ca44 [id=4fcf984156584e30aac26283f71adda2])
I20261118 08:54:56.730571 2880 tablet_loader.cc:96] loaded metadata for tablet 48a6e2d96fe442b6a84b4773993b4f54 (table default.table_567d34233a4b440a952301f19a6bcb10 [id=39ea4f28a4744d1290fe1e9246789b0a])
I20261118 08:54:56.730955 2880 tablet_loader.cc:96] loaded metadata for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf (table default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8])
I20261118 08:54:56.731254 2880 tablet_loader.cc:96] loaded metadata for tablet 50744c2d59124a22ac0a76f2f8e60974 (table default.table_294f03c6875c4da0845888273ea80a2a [id=0146c51b5d004bd6b63f5e1c9e1dcf30])
I20261118 08:54:56.731870 2880 tablet_loader.cc:96] loaded metadata for tablet 748947f292334e7a97abfa599634006d (table default.table_c3f2df1b938946c19c1ca00d8fcbd640 [id=812c3949fc5544c09b17567b3e81977b])
I20261118 08:54:56.731993 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "78d4026bc230421ea10b8c68a9ef7871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } } }
I20261118 08:54:56.732201 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [sys.catalog]: This master's current role is: FOLLOWER
I20261118 08:54:56.732506 2880 tablet_loader.cc:96] loaded metadata for tablet 86bf01af75674f5491b024ac43a063b0 (table default.table_5f1fb0758cf1413eaed1d8b4717aa902 [id=c2ea63be7ece4af4bcfb932bd174e957])
I20261118 08:54:56.733342 2880 tablet_loader.cc:96] loaded metadata for tablet d2ce64bc39634b5882b1beaa68e66cbe (table default.table_41d62c0cf6b54938b8e1770b859e56e8 [id=a57cd9d759e94822ae1e9ecb7faad9f7])
I20261118 08:54:56.733637 2880 tablet_loader.cc:96] loaded metadata for tablet d546b7f20a7f4d3b9e5612080df04be6 (table default.table_ef26d2c593994e118ff3d01a0f5a9a1d [id=19b3ec381a984304badca62634eadd70])
I20261118 08:54:56.734351 2880 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261118 08:54:56.735373 2880 catalog_manager.cc:1310] Loaded cluster ID: 0033b87516ad4c4a82378eedc366b8d8
I20261118 08:54:56.735484 2880 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261118 08:54:56.736742 2880 catalog_manager.cc:1555] Loading token signing keys...
I20261118 08:54:56.737569 2880 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Loaded TSK: 0
I20261118 08:54:56.738339 2880 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261118 08:54:56.739204 2880 catalog_manager.cc:1578] Loading latest processed Hive Metastore notification log event ID...
I20261118 08:54:56.740087 2880 catalog_manager.cc:6070] Last processed Hive Metastore notification event ID: -1
I20261118 08:54:56.762379 2804 master_service.cc:965] ReplaceTablet: received request to replace tablet 3e292319f0c44e34b49ae6d579df2ecb from {username='slave'} at 127.0.0.1:38312
I20261118 08:54:56.762530 2803 master_service.cc:965] ReplaceTablet: received request to replace tablet 3e292319f0c44e34b49ae6d579df2ecb from {username='slave'} at 127.0.0.1:38312
I20261118 08:54:56.762781 2801 master_service.cc:965] ReplaceTablet: received request to replace tablet 48a6e2d96fe442b6a84b4773993b4f54 from {username='slave'} at 127.0.0.1:38312
I20261118 08:54:56.762859 2805 master_service.cc:965] ReplaceTablet: received request to replace tablet 48a6e2d96fe442b6a84b4773993b4f54 from {username='slave'} at 127.0.0.1:38312
I20261118 08:54:56.763944 2803 catalog_manager.cc:2548] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38312:
table { table_name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff" } modify_external_catalogs: true
I20261118 08:54:56.764132 2802 master_service.cc:965] ReplaceTablet: received request to replace tablet 4eecc49bfe014d1d86c84e8a4f16a5bf from {username='slave'} at 127.0.0.1:38312
I20261118 08:54:56.764181 2797 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38312:
table { table_name: "default.Table_41d62c0cf6b54938b8e1770b859e56e8" } new_table_name: "default.Table_eefa15fa899c4b1e9162efce0db7e4af" modify_external_catalogs: true
I20261118 08:54:56.764325 2803 catalog_manager.cc:2796] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38312:
table { table_name: "default.Table_d2d73a819dc64fe0badbe6d84293bfff" } modify_external_catalogs: true
I20261118 08:54:56.764467 2796 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38312:
table { table_name: "default.Table_5f1fb0758cf1413eaed1d8b4717aa902" } new_table_name: "default.Table_d9954c97f0054586ab7cf9e5e98b3fa4" modify_external_catalogs: true
I20261118 08:54:56.763722 2804 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38312:
name: "default.Table_8f6feb81c1374d42bc1e0acb856b75cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:56.765012 2798 catalog_manager.cc:3585] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38312:
table { table_name: "default.Table_bc1f66261ac64a87912e19a32ce0ca44" } new_table_name: "default.Table_7fe6a97ebd294ce68601eaf9f252d0ee" modify_external_catalogs: true
I20261118 08:54:56.765517 2800 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38312:
name: "default.Table_82d1541f550a4d5482cff56810c97a22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:56.763718 2799 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38312:
name: "default.Table_0327d29a2f19450b872c54c554d7654e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20261118 08:54:56.772107 2860 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 10 received events: NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_bc1f66261ac64a87912e19a32ce0ca44, message={"server":"","servicePrincipal":"","db":"default","table":"table_bc1f66261ac64a87912e19a32ce0ca44","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_bc1f66261ac64a87912e19a32ce0ca44\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_bc1f66261ac64a87912e19a32ce0ca44\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_bc1f66261ac64a87912e19a32ce0ca44\",\"kudu.table_id\":\"4fcf984156584e30aac26283f71adda2\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1794992065, eventType=CREATE_TABLE, dbName=default, tableName=table_8b08d3e4ed824f1680377a6d5e7518a8, message={"server":"","servicePrincipal":"","db":"default","table":"table_8b08d3e4ed824f1680377a6d5e7518a8","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1794992065},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/warehouse/table_8b08d3e4ed824f1680377a6d5e7518a8\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_8b08d3e4ed824f1680377a6d5e7518a8\",\"kudu.table_id\":\"b42eb739941947f8b7d79fddd923eb52\",\"transient_lastDdlTime\":\"1794992065\",\"comment\":\"\",\"kudu.master_addresses\":\"127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"0033b87516ad4c4a82378eedc366b8d8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1794992065,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
@ 0x7fc67a688d5d google::LogMessage::Fail() at ??:0
@ 0x7fc67a68cb23 google::LogMessage::SendToLog() at ??:0
@ 0x7fc67a68875c google::LogMessage::Flush() at ??:0
I20261118 08:54:56.779068 2802 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 4eecc49bfe014d1d86c84e8a4f16a5bf
@ 0x7fc67a6889a1 google::LogMessage::~LogMessage() at ??:0
I20261118 08:54:56.781347 2802 catalog_manager.cc:6936] ReplaceTablet: tablet 4eecc49bfe014d1d86c84e8a4f16a5bf deleted and replaced by tablet 380fa8ff3e2f428b81b7567bc7bcefee
@ 0x7fc68fb089a4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20261118 08:54:56.801615 2801 catalog_manager.cc:6135] T 00000000000000000000000000000000 P 78d4026bc230421ea10b8c68a9ef7871: Sending DeleteTablet for 3 replicas of tablet 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:54:56.802258 2801 catalog_manager.cc:6936] ReplaceTablet: tablet 48a6e2d96fe442b6a84b4773993b4f54 deleted and replaced by tablet 625ed666c1f748a2a6fd4fbb44746496
@ 0x7fc68fb07251 kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fc68fb0eecc kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20261118 08:54:56.813431 915 tablet_service.cc:1547] Processing DeleteTablet for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf with delete_type TABLET_DATA_DELETED (replaced by tablet 380fa8ff3e2f428b81b7567bc7bcefee) from {username='slave'} at 127.0.0.1:55682
I20261118 08:54:56.813871 916 tablet_service.cc:1547] Processing DeleteTablet for tablet 48a6e2d96fe442b6a84b4773993b4f54 with delete_type TABLET_DATA_DELETED (replaced by tablet 625ed666c1f748a2a6fd4fbb44746496) from {username='slave'} at 127.0.0.1:55682
I20261118 08:54:56.815179 1053 tablet_service.cc:1547] Processing DeleteTablet for tablet 4eecc49bfe014d1d86c84e8a4f16a5bf with delete_type TABLET_DATA_DELETED (replaced by tablet 380fa8ff3e2f428b81b7567bc7bcefee) from {username='slave'} at 127.0.0.1:45156
I20261118 08:54:56.815277 1054 tablet_service.cc:1547] Processing DeleteTablet for tablet 48a6e2d96fe442b6a84b4773993b4f54 with delete_type TABLET_DATA_DELETED (replaced by tablet 625ed666c1f748a2a6fd4fbb44746496) from {username='slave'} at 127.0.0.1:45156
I20261118 08:54:56.819032 2802 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38312:
name: "default.Table_bfd124a61a904e60bc99bee8716f64cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7fc68fb0ed49 std::_Function_handler<>::_M_invoke() at ??:0
I20261118 08:54:56.819875 2805 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38312:
name: "default.Table_52bba9536ae942a59235d8087d61576b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x372c7d std::function<>::operator()() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:115
I20261118 08:54:56.823089 2801 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38312:
name: "default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261118 08:54:56.824566 2898 tablet_replica.cc:333] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:56.824776 2898 raft_consensus.cc:2261] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:56.825012 2898 raft_consensus.cc:2290] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fc67bad1dc9 kudu::Thread::SuperviseThread() at ??:0
I20261118 08:54:56.826103 2896 tablet_replica.cc:333] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:56.826390 2896 raft_consensus.cc:2261] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:56.826634 2896 raft_consensus.cc:2290] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:56.827246 2896 ts_tablet_manager.cc:1962] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fc67f1ac6db start_thread at ??:0
I20261118 08:54:56.829713 2898 ts_tablet_manager.cc:1962] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fc677c1c61f clone at ??:0
I20261118 08:54:56.829880 1053 tablet_service.cc:1500] Processing CreateTablet for tablet 380fa8ff3e2f428b81b7567bc7bcefee (DEFAULT_TABLE table=default.table_d2d73a819dc64fe0badbe6d84293bfff [id=fa8c2c5f6be14649a1bd9abaca0167c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20261118 08:54:56.830518 1053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 380fa8ff3e2f428b81b7567bc7bcefee. 1 dirs total, 0 dirs full, 0 dirs failed
I20261118 08:54:56.835709 2896 ts_tablet_manager.cc:1975] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:56.835860 2896 log.cc:1199] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/4eecc49bfe014d1d86c84e8a4f16a5bf
I20261118 08:54:56.836304 2896 ts_tablet_manager.cc:1996] T 4eecc49bfe014d1d86c84e8a4f16a5bf P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:56.837409 2898 ts_tablet_manager.cc:1975] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:56.837527 2898 log.cc:1199] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/4eecc49bfe014d1d86c84e8a4f16a5bf
I20261118 08:54:56.837800 2896 tablet_replica.cc:333] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: stopping tablet replica
I20261118 08:54:56.837947 2898 ts_tablet_manager.cc:1996] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:56.838035 2896 raft_consensus.cc:2261] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 1 LEADER]: Raft consensus shutting down.
I20261118 08:54:56.838387 2896 raft_consensus.cc:2290] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:56.839061 2896 ts_tablet_manager.cc:1962] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:56.839090 2898 tablet_replica.cc:333] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: stopping tablet replica
I20261118 08:54:56.839286 2898 raft_consensus.cc:2261] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261118 08:54:56.839473 2898 raft_consensus.cc:2290] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261118 08:54:56.840044 2898 ts_tablet_manager.cc:1962] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20261118 08:54:56.845880 2896 ts_tablet_manager.cc:1975] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:56.846045 2896 log.cc:1199] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-0/wal/wals/48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:54:56.846503 2896 ts_tablet_manager.cc:1996] T 48a6e2d96fe442b6a84b4773993b4f54 P ad2789ff5b54489c84966cff78be4721: Deleting consensus metadata
I20261118 08:54:56.847316 2900 tablet_bootstrap.cc:492] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042: Bootstrap starting.
I20261118 08:54:56.848860 2898 ts_tablet_manager.cc:1975] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20261118 08:54:56.848987 2898 log.cc:1199] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: Deleting WAL directory at /tmp/dist-test-task3eSafg/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1794992047095830-290-0/minicluster-data/ts-1/wal/wals/48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:54:56.849325 2900 tablet_bootstrap.cc:654] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042: Neither blocks nor log segments found. Creating new log.
I20261118 08:54:56.849431 2898 ts_tablet_manager.cc:1996] T 48a6e2d96fe442b6a84b4773993b4f54 P 213b5f7a64e74b109a3a9387e5975042: Deleting consensus metadata
I20261118 08:54:56.851269 2900 tablet_bootstrap.cc:492] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042: No bootstrap required, opened a new log
I20261118 08:54:56.851366 2900 ts_tablet_manager.cc:1449] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20261118 08:54:56.852097 2900 raft_consensus.cc:360] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:56.852237 2900 raft_consensus.cc:386] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261118 08:54:56.852293 2900 raft_consensus.cc:748] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213b5f7a64e74b109a3a9387e5975042, State: Initialized, Role: FOLLOWER
I20261118 08:54:56.852483 2900 consensus_queue.cc:261] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:56.854107 2900 ts_tablet_manager.cc:1480] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20261118 08:54:56.856462 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 1: this message will repeat every 5th retry.
W20261118 08:54:56.913125 1124 tablet.cc:2441] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261118 08:54:56.952610 1257 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
W20261118 08:54:56.999491 1009 connection.cc:573] server connection from 127.0.0.1:45156 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:56.999507 873 connection.cc:573] server connection from 127.0.0.1:55682 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:56.999794 1145 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:56.999658 2895 negotiation.cc:336] Failed RPC negotiation. Trace:
1118 08:54:56.810971 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.0.72.131:46447)
1118 08:54:56.812065 (+ 1094us) server_negotiation.cc:207] Beginning negotiation
1118 08:54:56.812072 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1118 08:54:56.812102 (+ 30us) server_negotiation.cc:408] Connection header received
1118 08:54:56.812200 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1118 08:54:56.812207 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1118 08:54:56.812292 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1118 08:54:56.812571 (+ 279us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1118 08:54:56.816151 (+ 3580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1118 08:54:56.820721 (+ 4570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1118 08:54:56.999599 (+178878us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44234: BlockingRecv error: recv error from unknown peer: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":283,"threads_started":1}
W20261118 08:54:57.000103 1010 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:57.000169 1264 connection.cc:573] client connection to 127.0.72.190:42341 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261118 08:54:57.000890 1264 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.72.190:42341)
W20261118 08:54:57.001286 1264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:57.001581 1264 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:57.001785 1264 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.72.190:42341)
W20261118 08:54:57.130183 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 1: this message will repeat every 5th retry.
I20261118 08:54:57.294582 984 heartbeater.cc:500] Master 127.0.72.190:42341 was elected leader, sending a full tablet report...
I20261118 08:54:57.324908 2914 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:57.325142 2914 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:57.325933 2914 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:57.328076 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:57.541000 2911 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:54:57.541224 2911 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:57.541880 2911 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:54:57.543037 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:54:57.543243 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:54:57.543370 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:57.543655 2911 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:54:58.164122 2922 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:58.164320 2922 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:58.164953 2922 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:54:58.165958 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:54:58.166116 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:54:58.166230 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:58.166474 2922 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:54:58.339927 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:58.340075 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:58.340684 2930 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:54:58.340719 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:54:58.340936 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:54:58.579646 290 master-stress-test.cc:579] Killing test thread 1/25
W20261118 08:54:58.765995 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 11: this message will repeat every 5th retry.
W20261118 08:54:58.925338 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 11: this message will repeat every 5th retry.
I20261118 08:54:58.970541 2935 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:58.970737 2935 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:58.971525 2935 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:58.972391 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20261118 08:54:58.973538 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:54:58.973683 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:54:58.973835 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:58.974081 2935 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:54:59.513338 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 11: this message will repeat every 5th retry.
I20261118 08:54:59.533136 2922 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:54:59.533301 2922 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:54:59.534049 2922 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:54:59.534900 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:54:59.535069 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:54:59.535203 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:54:59.535462 2922 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:54:59.625959 2943 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:54:59.626154 2943 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:54:59.626849 2943 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:54:59.628777 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:54:59.628944 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:54:59.629087 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:54:59.629323 2943 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:54:59.872606 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 11: this message will repeat every 5th retry.
I20261118 08:55:00.480723 2924 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:00.480903 2924 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:55:00.481573 2924 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:55:00.482386 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:55:00.482545 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:55:00.482679 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:00.482923 2924 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:00.958114 2956 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:00.958333 2956 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:00.959080 2956 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:00.961369 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:55:00.961509 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:55:00.961660 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:00.961908 2956 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:55:01.603644 2967 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:55:01.603856 2967 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:01.604725 2967 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:55:01.605660 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:55:01.605860 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:55:01.606006 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:01.606235 2967 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:55:02.074290 2970 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:02.074482 2970 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:02.075208 2970 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:02.077728 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:55:02.077939 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:55:02.078099 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:02.078395 2970 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:55:03.289548 2923 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:03.289760 2923 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:03.290491 2923 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:03.291337 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:55:03.291550 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:55:03.291647 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:03.291880 2923 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:03.342417 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:03.342625 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:55:04.198098 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 11: this message will repeat every 5th retry.
W20261118 08:55:05.403669 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 11: this message will repeat every 5th retry.
I20261118 08:55:05.482529 3014 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:05.482851 3014 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:05.483740 3014 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:05.484767 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20261118 08:55:05.485849 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:55:05.486025 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:55:05.486156 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:05.486399 3014 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:55:05.703622 3013 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:55:05.703832 3013 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:05.704566 3013 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:55:05.705437 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:55:05.705590 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:55:05.705739 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:05.705971 3013 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261118 08:55:07.225171 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 6: this message will repeat every 5th retry.
W20261118 08:55:07.998588 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 16: this message will repeat every 5th retry.
W20261118 08:55:08.118414 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 6: this message will repeat every 5th retry.
I20261118 08:55:08.344391 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:08.344623 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:55:09.087571 3009 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:09.087778 3009 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:09.088557 3009 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:09.089561 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:55:09.089715 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:55:09.089874 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:09.090145 3009 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:09.165706 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 16: this message will repeat every 5th retry.
W20261118 08:55:09.360040 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 16: this message will repeat every 5th retry.
W20261118 08:55:09.896785 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 16: this message will repeat every 5th retry.
I20261118 08:55:09.960031 3051 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:09.960263 3051 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:09.961004 3051 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:09.963049 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:55:09.963208 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:55:09.963357 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:09.963650 3051 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:55:10.621586 3009 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:10.621821 3009 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:10.622414 3009 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:10.623314 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:55:10.623510 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:55:10.623628 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:10.623854 3009 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:12.581561 3076 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:55:12.581813 3076 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:12.582634 3076 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:55:12.583623 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:55:12.583786 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:55:12.583930 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:12.584182 3076 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:55:12.617372 3077 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:12.617573 3077 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:12.618295 3077 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:12.619261 2639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 08:55:12.620399 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:55:12.620559 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:55:12.620682 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:12.620905 3077 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:55:12.704136 3009 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:12.704350 3009 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:55:12.705029 3009 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:55:12.706007 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:55:12.706167 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:55:12.706311 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:12.706550 3009 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:13.345884 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:13.346053 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:55:13.436771 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 16: this message will repeat every 5th retry.
I20261118 08:55:14.636415 3009 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:14.636583 3009 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:14.637305 3009 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:14.638192 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:55:14.638337 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:55:14.638448 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:14.638674 3009 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:14.873409 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 16: this message will repeat every 5th retry.
I20261118 08:55:16.674403 3110 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:16.674600 3110 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:55:16.675334 3110 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:55:16.676064 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:55:16.676261 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:55:16.676383 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:16.676620 3110 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:16.852854 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 11: this message will repeat every 5th retry.
W20261118 08:55:17.424782 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 11: this message will repeat every 5th retry.
W20261118 08:55:18.037348 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 21: this message will repeat every 5th retry.
I20261118 08:55:18.347512 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:18.347718 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:55:19.310201 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 21: this message will repeat every 5th retry.
W20261118 08:55:19.495365 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 21: this message will repeat every 5th retry.
W20261118 08:55:20.152040 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 21: this message will repeat every 5th retry.
I20261118 08:55:21.214655 3147 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:55:21.214861 3147 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:21.215817 3147 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:55:21.216738 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:55:21.216929 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:55:21.217072 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:21.217336 3147 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:55:22.421103 3158 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:22.421324 3158 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:22.422132 3158 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:22.423287 2639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:55:22.424746 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:55:22.424921 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:55:22.425053 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:22.425354 3158 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:55:23.290721 2998 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:23.290930 2998 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:23.291654 2998 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:23.292619 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:55:23.292770 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:55:23.292913 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:23.293151 2998 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:23.349212 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:23.349423 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:55:23.535778 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 21: this message will repeat every 5th retry.
W20261118 08:55:25.075994 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 21: this message will repeat every 5th retry.
W20261118 08:55:25.620873 1256 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (343 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:55:25.695674 1122 heartbeater.cc:648] Failed to heartbeat to 127.0.72.190:42341 (72 consecutive failures): Network error: Failed to ping master at 127.0.72.190:42341: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:55:25.846400 983 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (146 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:55:26.376350 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 16: this message will repeat every 5th retry.
I20261118 08:55:26.647089 3110 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:26.647313 3110 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:26.648067 3110 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:26.648962 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:55:26.649140 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:55:26.649291 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:26.649569 3110 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:26.769910 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 16: this message will repeat every 5th retry.
W20261118 08:55:27.285463 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 26: this message will repeat every 5th retry.
I20261118 08:55:28.350706 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:28.350883 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:55:28.431332 3110 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:28.431565 3110 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:28.432268 3110 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:28.433226 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:55:28.433385 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:55:28.433503 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:28.433722 3110 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:29.222297 3220 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:29.222513 3220 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:29.223501 3220 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:29.224879 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:55:29.226145 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:55:29.226372 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:55:29.226468 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:29.226708 3220 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:55:29.710383 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 26: this message will repeat every 5th retry.
W20261118 08:55:30.112123 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 26: this message will repeat every 5th retry.
W20261118 08:55:30.612264 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 26: this message will repeat every 5th retry.
I20261118 08:55:31.589365 3182 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:31.589563 3182 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:31.590410 3182 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:31.591483 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:55:31.591630 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:55:31.591786 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:31.592043 3182 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:33.352298 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:33.352494 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:55:33.658567 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 26: this message will repeat every 5th retry.
I20261118 08:55:34.941478 3242 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:34.941672 3242 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:55:34.942396 3242 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:55:34.943277 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:55:34.943454 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:55:34.943619 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:34.943938 3242 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:55:35.528959 3242 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:35.529157 3242 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:35.529842 3242 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:35.530794 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:55:35.530956 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:55:35.531064 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:35.531307 3242 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:35.652654 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 26: this message will repeat every 5th retry.
I20261118 08:55:36.319967 3285 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:55:36.320152 3285 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:36.320981 3285 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:55:36.321956 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:55:36.322520 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:55:36.322631 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:36.322913 3285 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261118 08:55:36.791501 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 21: this message will repeat every 5th retry.
W20261118 08:55:36.938033 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 21: this message will repeat every 5th retry.
W20261118 08:55:37.462366 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 31: this message will repeat every 5th retry.
I20261118 08:55:38.354204 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:38.354455 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:55:39.781598 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 31: this message will repeat every 5th retry.
W20261118 08:55:39.964975 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 31: this message will repeat every 5th retry.
W20261118 08:55:41.245625 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 31: this message will repeat every 5th retry.
I20261118 08:55:42.528915 3333 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:42.529117 3333 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:42.530346 3333 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:42.531229 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:55:42.532398 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:55:42.532545 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:55:42.532692 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:42.532941 3333 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:55:42.811623 3332 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:42.811825 3332 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:42.812507 3332 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:42.813449 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:55:42.813611 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:55:42.813727 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:42.813973 3332 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:43.195899 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 31: this message will repeat every 5th retry.
I20261118 08:55:43.355970 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:43.356166 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:55:45.480193 3332 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:45.480386 3332 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:45.481169 3332 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:45.481971 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:55:45.482698 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:55:45.482836 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:45.483126 3332 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:45.683813 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 31: this message will repeat every 5th retry.
W20261118 08:55:46.574167 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 26: this message will repeat every 5th retry.
W20261118 08:55:47.035578 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 26: this message will repeat every 5th retry.
W20261118 08:55:47.488520 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 36: this message will repeat every 5th retry.
I20261118 08:55:48.357723 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:48.357944 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:55:48.801263 3386 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:55:48.801452 3386 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:48.802232 3386 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:55:48.803275 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:55:48.803471 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:55:48.803624 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:48.803907 3386 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:55:49.107906 3252 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:49.108129 3252 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:49.109066 3252 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:49.109977 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:55:49.110144 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:55:49.110285 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:49.110561 3252 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:49.359207 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 36: this message will repeat every 5th retry.
W20261118 08:55:49.925786 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 36: this message will repeat every 5th retry.
W20261118 08:55:51.284401 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 36: this message will repeat every 5th retry.
I20261118 08:55:51.857213 3182 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:51.857421 3182 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:55:51.858119 3182 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:55:51.859035 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:55:51.859642 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:55:51.859743 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:51.860023 3182 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:53.033164 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 36: this message will repeat every 5th retry.
I20261118 08:55:53.359467 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:53.359680 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:55:53.612264 3252 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:55:53.612531 3252 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:55:53.613265 3252 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:55:53.614245 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:55:53.614434 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:55:53.614563 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:55:53.615234 3252 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:55:55.629081 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 36: this message will repeat every 5th retry.
W20261118 08:55:56.517138 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 31: this message will repeat every 5th retry.
W20261118 08:55:56.662532 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 31: this message will repeat every 5th retry.
W20261118 08:55:58.249585 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 41: this message will repeat every 5th retry.
I20261118 08:55:58.361294 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:55:58.361491 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:55:59.317359 3479 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:55:59.317552 3479 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:55:59.318473 3479 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:55:59.319428 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:55:59.320870 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:55:59.321014 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:55:59.321136 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:55:59.321372 3479 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:56:00.030388 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 41: this message will repeat every 5th retry.
W20261118 08:56:00.670131 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 41: this message will repeat every 5th retry.
W20261118 08:56:01.363981 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 41: this message will repeat every 5th retry.
I20261118 08:56:02.567337 3512 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:56:02.567566 3512 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:56:02.568261 3512 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:56:02.570469 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:56:02.570636 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:56:02.570753 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:56:02.570984 3512 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:56:03.363133 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:03.363415 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:03.619737 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 41: this message will repeat every 5th retry.
I20261118 08:56:04.696527 3468 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:04.696734 3468 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:04.697592 3468 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:04.698611 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:56:04.698777 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:56:04.698912 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:04.699177 3468 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:56:05.223538 3469 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:05.223729 3469 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:05.224434 3469 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:05.225507 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:56:05.226583 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:56:05.226745 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:05.227041 3507 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:05.329495 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 41: this message will repeat every 5th retry.
W20261118 08:56:06.033017 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 36: this message will repeat every 5th retry.
W20261118 08:56:07.077019 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 36: this message will repeat every 5th retry.
I20261118 08:56:07.303799 3556 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:56:07.304013 3556 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:56:07.304716 3556 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:56:07.305799 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 08:56:07.306924 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:56:07.307057 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:56:07.307192 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:56:07.307471 3556 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:56:07.475507 3554 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:56:07.476145 3554 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:07.477133 3554 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:56:07.478021 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:56:07.478178 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:56:07.478279 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:07.478523 3554 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:56:08.364980 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:08.365206 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:08.377190 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 46: this message will repeat every 5th retry.
I20261118 08:56:09.102692 3551 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:09.102882 3551 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:09.103664 3551 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:09.104684 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:56:09.104856 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:56:09.104946 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:09.105177 3551 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:56:09.575156 3551 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:09.575352 3551 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:56:09.575985 3551 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:56:09.576920 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:56:09.577099 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:56:09.577193 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:09.577422 3551 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:10.280453 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 46: this message will repeat every 5th retry.
W20261118 08:56:10.542582 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 46: this message will repeat every 5th retry.
W20261118 08:56:10.976244 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 46: this message will repeat every 5th retry.
I20261118 08:56:13.107116 3468 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:13.107352 3468 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:13.109134 3468 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:13.109826 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:56:13.109995 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:56:13.110143 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:13.110420 3468 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:56:13.366639 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:13.366827 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:13.676385 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 46: this message will repeat every 5th retry.
I20261118 08:56:15.253789 3619 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:56:15.254024 3619 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:56:15.254783 3619 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:56:15.255825 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:56:15.257071 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:56:15.257243 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:56:15.257349 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:56:15.257596 3619 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:56:15.387138 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 46: this message will repeat every 5th retry.
W20261118 08:56:15.668114 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 41: this message will repeat every 5th retry.
W20261118 08:56:17.322515 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 41: this message will repeat every 5th retry.
I20261118 08:56:17.717067 3468 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:17.717267 3468 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:56:17.717990 3468 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:56:17.718888 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:56:17.719050 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:56:17.719187 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:17.719466 3468 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:56:18.369143 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:18.369321 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:18.986331 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 51: this message will repeat every 5th retry.
W20261118 08:56:20.174638 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 51: this message will repeat every 5th retry.
W20261118 08:56:20.454113 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 51: this message will repeat every 5th retry.
I20261118 08:56:21.132073 3551 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:21.132306 3551 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:21.133185 3551 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:21.133998 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:56:21.134179 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:56:21.134320 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:21.134579 3551 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:21.345757 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 51: this message will repeat every 5th retry.
I20261118 08:56:23.370868 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:23.371068 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:24.701244 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 51: this message will repeat every 5th retry.
W20261118 08:56:24.821265 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 51: this message will repeat every 5th retry.
W20261118 08:56:25.555078 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 46: this message will repeat every 5th retry.
W20261118 08:56:25.914522 1256 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (765 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:56:25.994333 983 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (266 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:56:26.147709 1121 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (312 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:56:26.618682 3708 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:56:26.618866 3708 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:26.619747 3708 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:56:26.620697 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:56:26.620852 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:56:26.621023 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:26.621294 3708 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261118 08:56:27.159123 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 46: this message will repeat every 5th retry.
I20261118 08:56:28.372611 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:28.372826 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:29.008963 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 56: this message will repeat every 5th retry.
I20261118 08:56:29.925711 3748 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:29.925987 3748 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:29.926647 3748 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:29.927673 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:56:29.927856 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:56:29.927985 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:29.928287 3748 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:29.930732 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 56: this message will repeat every 5th retry.
I20261118 08:56:30.124997 3754 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:56:30.125273 3754 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:56:30.126106 3754 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:56:30.127000 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:56:30.128423 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:56:30.128595 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:56:30.128767 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:56:30.129030 3754 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:56:30.750974 3748 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:30.751205 3748 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:30.751905 3748 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:30.752823 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:56:30.753000 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:56:30.753130 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:30.753372 3748 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:30.894093 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 56: this message will repeat every 5th retry.
W20261118 08:56:30.950435 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 56: this message will repeat every 5th retry.
I20261118 08:56:30.957609 3764 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:56:30.957772 3764 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:30.958381 3764 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:56:30.959302 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:56:30.959477 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:56:30.959620 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:30.959905 3764 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:56:33.374001 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:33.374167 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:33.728518 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 56: this message will repeat every 5th retry.
I20261118 08:56:34.067198 3748 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:34.067408 3748 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:56:34.068578 3748 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:56:34.069060 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:56:34.069252 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:56:34.069344 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:34.069576 3748 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:35.507803 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 56: this message will repeat every 5th retry.
I20261118 08:56:35.747218 3748 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:35.747476 3748 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:35.748227 3748 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:35.749169 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:56:35.749332 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:56:35.749476 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:35.749755 3748 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:35.853072 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 51: this message will repeat every 5th retry.
W20261118 08:56:36.423677 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 51: this message will repeat every 5th retry.
I20261118 08:56:38.376693 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:38.376899 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:39.576707 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 61: this message will repeat every 5th retry.
W20261118 08:56:40.147802 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 61: this message will repeat every 5th retry.
W20261118 08:56:40.526678 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 61: this message will repeat every 5th retry.
W20261118 08:56:40.870857 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 61: this message will repeat every 5th retry.
I20261118 08:56:42.543140 3749 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:42.543349 3749 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:42.544258 3749 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:42.545132 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:56:42.545733 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:56:42.545837 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:42.546088 3749 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:43.245555 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 61: this message will repeat every 5th retry.
I20261118 08:56:43.348345 3813 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:43.348552 3813 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:56:43.349749 3813 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:56:43.350172 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:56:43.350325 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:56:43.350476 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:43.350739 3813 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:56:43.378424 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:43.378635 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:45.004976 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 56: this message will repeat every 5th retry.
W20261118 08:56:45.552264 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 61: this message will repeat every 5th retry.
I20261118 08:56:46.783135 3886 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:56:46.783317 3886 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:46.784143 3886 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:56:46.785048 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:56:46.785250 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:56:46.785362 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:46.785617 3886 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261118 08:56:46.916731 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 56: this message will repeat every 5th retry.
I20261118 08:56:48.380386 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:48.380594 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:56:48.998595 3906 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:56:48.998813 3906 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:56:48.999770 3906 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:56:49.004200 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:56:49.005437 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:56:49.005641 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:56:49.005762 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:56:49.006021 3906 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:56:49.719944 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 66: this message will repeat every 5th retry.
I20261118 08:56:49.774554 3918 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:49.774799 3918 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:56:49.775539 3918 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:56:49.776458 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:56:49.776613 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:56:49.776749 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:49.776984 3918 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:50.391311 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 66: this message will repeat every 5th retry.
I20261118 08:56:50.430733 3918 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:50.430938 3918 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:50.431639 3918 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:50.432526 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:56:50.432701 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:56:50.432807 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:50.433034 3918 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:56:50.691210 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 66: this message will repeat every 5th retry.
W20261118 08:56:51.021389 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 66: this message will repeat every 5th retry.
I20261118 08:56:52.601811 2644 maintenance_manager.cc:419] P 857a577b88ba46cc846ecd1a25a722ca: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033355
I20261118 08:56:52.662731 2643 maintenance_manager.cc:643] P 857a577b88ba46cc846ecd1a25a722ca: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.060s user 0.022s sys 0.004s Metrics: {"bytes_written":16764,"cfile_init":1,"dirs.queue_time_us":889,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":881,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":695,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":38,"thread_start_us":942,"threads_started":2,"wal-append.queue_time_us":330}
I20261118 08:56:52.664049 2644 maintenance_manager.cc:419] P 857a577b88ba46cc846ecd1a25a722ca: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1964 bytes on disk
I20261118 08:56:52.664755 2643 maintenance_manager.cc:643] P 857a577b88ba46cc846ecd1a25a722ca: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20261118 08:56:53.382496 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:53.382723 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:53.763945 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 66: this message will repeat every 5th retry.
W20261118 08:56:54.763553 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 61: this message will repeat every 5th retry.
W20261118 08:56:54.921329 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 66: this message will repeat every 5th retry.
W20261118 08:56:57.439637 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 61: this message will repeat every 5th retry.
I20261118 08:56:57.449100 3918 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:57.449283 3918 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:57.449903 3918 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:57.450910 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:56:57.451118 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:56:57.451258 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:57.452208 3813 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:56:58.224413 3918 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:56:58.224625 3918 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:56:58.225358 3918 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:56:58.226357 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:56:58.226532 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:56:58.226634 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:56:58.226876 3918 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:56:58.384636 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:56:58.384876 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:56:58.816382 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 71: this message will repeat every 5th retry.
W20261118 08:56:59.408268 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 71: this message will repeat every 5th retry.
W20261118 08:57:00.818118 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 71: this message will repeat every 5th retry.
W20261118 08:57:00.913568 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 71: this message will repeat every 5th retry.
I20261118 08:57:01.579334 4012 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:57:01.579571 4012 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:01.580274 4012 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:57:01.581115 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:57:01.581290 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:57:01.581383 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:01.581595 4012 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:57:02.811450 4029 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:57:02.811668 4029 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:57:02.812465 4029 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:57:02.813709 2639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:57:02.815439 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:57:02.815616 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:57:02.815765 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:57:02.816033 4029 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:57:03.386439 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:03.386622 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:03.730224 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 71: this message will repeat every 5th retry.
W20261118 08:57:04.846489 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 71: this message will repeat every 5th retry.
W20261118 08:57:05.071432 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 66: this message will repeat every 5th retry.
I20261118 08:57:06.426401 4049 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:06.426612 4049 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:57:06.427289 4049 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:57:06.428308 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:57:06.428484 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:57:06.428607 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:06.428861 4049 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:07.287979 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 66: this message will repeat every 5th retry.
I20261118 08:57:07.598213 4049 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:07.598405 4049 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:07.598994 4049 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:07.599918 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:57:07.600090 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:57:07.600194 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:07.600435 4049 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:07.792465 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 76: this message will repeat every 5th retry.
I20261118 08:57:08.388123 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:08.388317 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:08.585072 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 76: this message will repeat every 5th retry.
W20261118 08:57:10.655377 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 76: this message will repeat every 5th retry.
W20261118 08:57:10.749447 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 76: this message will repeat every 5th retry.
I20261118 08:57:13.389655 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:13.389851 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:13.496822 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 76: this message will repeat every 5th retry.
I20261118 08:57:13.831357 4050 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:13.831593 4050 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:57:13.832554 4050 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:57:13.833523 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:57:13.833685 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:57:13.833845 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:13.834079 4050 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:14.454798 3994 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:14.455018 3994 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:14.455883 3994 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:14.456077 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 76: this message will repeat every 5th retry.
W20261118 08:57:14.457000 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:57:14.457168 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:57:14.457310 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:14.457564 3994 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:14.797785 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 71: this message will repeat every 5th retry.
W20261118 08:57:16.898777 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 71: this message will repeat every 5th retry.
W20261118 08:57:17.596490 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 81: this message will repeat every 5th retry.
I20261118 08:57:18.391254 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:18.391487 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:18.687974 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 81: this message will repeat every 5th retry.
I20261118 08:57:19.389554 4231 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:19.389760 4231 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:19.390467 4231 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:19.391196 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:57:19.391350 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:57:19.391530 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:19.391784 4231 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:20.579126 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 81: this message will repeat every 5th retry.
W20261118 08:57:21.303167 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 81: this message will repeat every 5th retry.
I20261118 08:57:21.365702 4264 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:57:21.365886 4264 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:21.366784 4264 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:57:21.367770 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:57:21.367939 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:57:21.368062 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:21.368319 4264 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:57:21.828130 4273 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:57:21.828332 4273 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:57:21.829241 4273 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:57:21.830199 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:57:21.831758 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:57:21.831933 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:57:21.832042 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:57:21.832307 4273 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:57:22.138616 4231 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:22.138810 4231 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:22.139652 4231 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:22.140278 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:57:22.140488 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:57:22.140599 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:22.140807 4231 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:23.392838 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:23.393049 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:23.609048 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 81: this message will repeat every 5th retry.
W20261118 08:57:23.847486 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 81: this message will repeat every 5th retry.
I20261118 08:57:24.438290 4299 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:24.438506 4299 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:24.439153 4299 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:24.440070 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:57:24.440315 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:57:24.440454 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:24.441771 4299 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:25.136732 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 76: this message will repeat every 5th retry.
I20261118 08:57:25.658556 4313 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:25.658771 4313 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:57:25.659446 4313 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:57:25.660317 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:57:25.660490 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:57:25.660580 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:25.660792 4313 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:25.776654 4334 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:57:25.776886 4334 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:57:25.777635 4334 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:57:25.780184 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:57:25.780369 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:57:25.780481 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:57:25.780716 4334 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:57:25.946666 1256 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (1203 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:57:26.140319 983 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (386 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:57:26.180120 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 76: this message will repeat every 5th retry.
W20261118 08:57:26.488530 1122 heartbeater.cc:648] Failed to heartbeat to 127.0.72.190:42341 (372 consecutive failures): Network error: Failed to ping master at 127.0.72.190:42341: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:57:27.067734 4354 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:57:27.067942 4354 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:57:27.069057 4354 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:57:27.069917 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 08:57:27.071238 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:57:27.071425 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:57:27.071551 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:57:27.071816 4354 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:57:27.310225 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 86: this message will repeat every 5th retry.
W20261118 08:57:28.060025 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 86: this message will repeat every 5th retry.
I20261118 08:57:28.394341 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:28.394522 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:57:28.864213 4370 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:57:28.864427 4370 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:28.865150 4370 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:57:28.866175 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:57:28.866361 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:57:28.866454 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:28.866681 4370 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261118 08:57:29.992378 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 86: this message will repeat every 5th retry.
W20261118 08:57:30.968719 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 86: this message will repeat every 5th retry.
I20261118 08:57:32.826208 4314 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:32.826406 4314 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:32.827298 4314 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:32.828336 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:57:32.828920 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:57:32.829025 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:32.829272 4314 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:33.003051 4314 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:33.003265 4314 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:33.004040 4314 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:33.004946 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:57:33.005134 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:57:33.005229 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:33.005458 4314 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:33.395777 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:33.395953 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:34.058838 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 86: this message will repeat every 5th retry.
W20261118 08:57:34.159585 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 86: this message will repeat every 5th retry.
W20261118 08:57:35.202608 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 81: this message will repeat every 5th retry.
W20261118 08:57:35.974812 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 81: this message will repeat every 5th retry.
W20261118 08:57:36.735183 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 91: this message will repeat every 5th retry.
W20261118 08:57:38.089156 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 91: this message will repeat every 5th retry.
I20261118 08:57:38.375211 4314 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:38.375442 4314 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:57:38.376134 4314 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:57:38.377053 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:57:38.377205 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:57:38.377357 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:38.377610 4314 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:38.391023 4314 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:38.391206 4314 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:38.391891 4314 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:38.392889 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:57:38.393100 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:57:38.393255 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:38.393486 4314 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:38.397275 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:38.397423 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:40.281062 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 91: this message will repeat every 5th retry.
I20261118 08:57:41.064025 4313 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:41.064213 4313 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:41.064800 4313 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:41.065735 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:57:41.065914 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:57:41.066059 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:41.066287 4313 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:41.566818 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 91: this message will repeat every 5th retry.
I20261118 08:57:43.398764 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:43.398944 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:43.972656 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 91: this message will repeat every 5th retry.
W20261118 08:57:44.146862 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 91: this message will repeat every 5th retry.
I20261118 08:57:44.376641 4553 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:57:44.376856 4553 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:57:44.377750 4553 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:57:44.378757 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:57:44.380162 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:57:44.380307 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:57:44.380415 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:57:44.380642 4553 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:57:44.890688 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 86: this message will repeat every 5th retry.
I20261118 08:57:46.291494 4583 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:57:46.291733 4583 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:46.292814 4583 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:57:46.293938 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:57:46.294128 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:57:46.294243 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:46.294536 4583 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261118 08:57:46.431417 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 96: this message will repeat every 5th retry.
W20261118 08:57:46.655726 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 86: this message will repeat every 5th retry.
W20261118 08:57:48.297097 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 96: this message will repeat every 5th retry.
I20261118 08:57:48.400592 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:48.400816 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:57:48.736032 4540 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:48.736230 4540 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:57:48.737087 4540 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:57:48.738003 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:57:48.738160 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:57:48.738265 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:48.738526 4540 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:50.783576 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 96: this message will repeat every 5th retry.
W20261118 08:57:51.245422 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 96: this message will repeat every 5th retry.
I20261118 08:57:51.670264 4657 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:57:51.670480 4657 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:51.671205 4657 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:57:51.672215 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:57:51.672395 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:57:51.672531 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:51.673242 4657 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:57:52.485152 4681 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:57:52.485369 4681 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:57:52.486109 4681 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:57:52.487103 2639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:57:52.488272 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:57:52.488449 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:57:52.488554 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:57:52.488758 4681 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:57:53.160959 4678 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:53.161157 4678 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:53.161870 4678 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:53.162869 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:57:53.163060 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:57:53.163156 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:53.163399 4678 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:57:53.402053 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:53.402244 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:57:53.524145 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 96: this message will repeat every 5th retry.
W20261118 08:57:54.312764 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 96: this message will repeat every 5th retry.
W20261118 08:57:55.071861 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 91: this message will repeat every 5th retry.
W20261118 08:57:56.021706 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 91: this message will repeat every 5th retry.
W20261118 08:57:56.566620 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 101: this message will repeat every 5th retry.
I20261118 08:57:57.044891 4742 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:57:57.045120 4742 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:57:57.045958 4742 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:57:57.047039 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:57:57.047231 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:57:57.047358 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:57:57.047664 4742 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:57:58.063182 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 101: this message will repeat every 5th retry.
I20261118 08:57:58.404238 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:57:58.404412 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:58:00.476320 4741 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:00.476521 4741 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:00.477169 4741 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:00.478080 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:58:00.478230 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:58:00.478367 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:00.478607 4741 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:00.626058 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 101: this message will repeat every 5th retry.
W20261118 08:58:01.224153 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 101: this message will repeat every 5th retry.
I20261118 08:58:01.458575 4832 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:58:01.458774 4832 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:01.459609 4832 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:58:01.460556 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:58:01.460734 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:58:01.460877 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:01.461124 4832 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:58:01.837862 4838 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:01.838052 4838 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:01.838773 4838 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:01.839820 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:58:01.840929 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:01.841061 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:58:01.841203 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:01.841441 4838 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:58:03.405916 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:03.406118 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:03.720842 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 101: this message will repeat every 5th retry.
W20261118 08:58:03.870702 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 101: this message will repeat every 5th retry.
W20261118 08:58:04.514531 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 96: this message will repeat every 5th retry.
W20261118 08:58:05.857724 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 96: this message will repeat every 5th retry.
I20261118 08:58:05.903087 4896 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:05.903311 4896 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:58:05.904120 4896 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:58:05.904906 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:58:05.905090 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:58:05.905206 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:05.905431 4896 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:58:07.272198 4896 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:07.272421 4896 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:07.273038 4896 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:07.273949 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:58:07.274128 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:58:07.274224 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:07.274467 4896 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:58:07.442584 4931 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:58:07.442770 4931 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:07.443519 4931 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:58:07.444458 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:58:07.444646 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:58:07.444777 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:07.445013 4931 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20261118 08:58:07.616060 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 106: this message will repeat every 5th retry.
W20261118 08:58:07.771492 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 106: this message will repeat every 5th retry.
I20261118 08:58:08.407959 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:08.408164 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:58:09.211404 4932 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:09.211655 4932 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:09.212359 4932 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:09.213320 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:58:09.213482 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:58:09.213601 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:09.213835 4932 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:58:10.523654 4981 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:10.523870 4981 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:10.524664 4981 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:10.526247 2639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:58:10.527400 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:58:10.527592 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:58:10.527729 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:10.528002 4981 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:58:10.605207 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 106: this message will repeat every 5th retry.
W20261118 08:58:10.729740 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 106: this message will repeat every 5th retry.
I20261118 08:58:11.769147 5002 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:11.769343 5002 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:11.770032 5002 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:11.772464 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:58:11.772655 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:58:11.772756 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:11.772970 5002 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:58:13.239437 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 106: this message will repeat every 5th retry.
I20261118 08:58:13.409818 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:13.410010 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:58:13.869660 5040 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:58:13.869935 5040 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
W20261118 08:58:13.871131 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 106: this message will repeat every 5th retry.
I20261118 08:58:13.871647 5040 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:58:13.872478 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:58:13.872601 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:58:13.872759 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:13.873013 5040 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:58:13.914664 4933 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:13.914865 4933 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:13.915640 4933 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:13.916586 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:58:13.916769 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:58:13.916903 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:13.917186 4933 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:14.086374 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 101: this message will repeat every 5th retry.
I20261118 08:58:14.991521 5056 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:14.991725 5056 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:14.992527 5056 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:14.997365 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:58:14.997550 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:58:14.998112 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:14.998428 5056 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:58:15.795671 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 101: this message will repeat every 5th retry.
I20261118 08:58:15.996923 4933 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:15.997119 4933 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:58:15.997906 4933 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:58:15.998978 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:58:15.999141 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:58:15.999282 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:15.999564 4933 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:17.211984 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 111: this message will repeat every 5th retry.
W20261118 08:58:17.652159 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 111: this message will repeat every 5th retry.
I20261118 08:58:17.998585 5106 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:17.998780 5106 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:17.999498 5106 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:18.000538 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20261118 08:58:18.002205 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:58:18.002370 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:58:18.002485 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:18.002753 5106 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:58:18.411355 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:18.411602 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:20.220495 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 111: this message will repeat every 5th retry.
W20261118 08:58:21.214172 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 111: this message will repeat every 5th retry.
W20261118 08:58:23.025146 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 111: this message will repeat every 5th retry.
I20261118 08:58:23.413147 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:23.413333 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:23.712766 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 111: this message will repeat every 5th retry.
W20261118 08:58:24.020530 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 106: this message will repeat every 5th retry.
I20261118 08:58:25.256313 5216 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:25.256512 5216 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:25.257220 5216 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:25.258222 2639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:25.258247 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:58:25.259526 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:25.259686 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:58:25.259807 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:25.260017 5216 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:58:26.000677 1256 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (1630 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:26.029651 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 106: this message will repeat every 5th retry.
W20261118 08:58:26.308990 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 116: this message will repeat every 5th retry.
W20261118 08:58:26.317118 983 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (506 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:26.817726 1122 heartbeater.cc:648] Failed to heartbeat to 127.0.72.190:42341 (521 consecutive failures): Network error: Failed to ping master at 127.0.72.190:42341: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:58:26.827726 4933 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:26.827901 4933 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:26.828666 4933 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:26.829648 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:58:26.829793 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:58:26.829875 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:26.830077 4933 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:27.911135 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 116: this message will repeat every 5th retry.
I20261118 08:58:28.414810 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:28.415010 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:30.234118 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 116: this message will repeat every 5th retry.
I20261118 08:58:31.916823 4896 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:31.917011 4896 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:58:31.917696 4896 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:58:31.918629 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:58:31.918766 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:58:31.918849 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:31.919044 4896 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:32.000252 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 116: this message will repeat every 5th retry.
I20261118 08:58:32.294178 4896 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:32.294350 4896 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:32.294960 4896 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:32.295940 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:58:32.296077 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:58:32.296203 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:32.296473 4896 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:58:32.575585 4896 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:32.575778 4896 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:32.576421 4896 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:32.577317 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:58:32.577447 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:58:32.577556 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:32.577790 4896 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:32.589865 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 116: this message will repeat every 5th retry.
I20261118 08:58:32.875303 5331 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:32.875540 5331 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:32.876706 5331 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:32.877676 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:58:32.878756 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:32.878897 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:58:32.879011 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:32.879268 5331 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:58:33.396426 5341 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:58:33.396607 5341 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:33.397483 5341 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:58:33.398399 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:58:33.398571 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:58:33.398659 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:33.398870 5341 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:58:33.416570 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:33.416781 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:33.618379 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 116: this message will repeat every 5th retry.
W20261118 08:58:34.010331 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 111: this message will repeat every 5th retry.
W20261118 08:58:35.366449 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 111: this message will repeat every 5th retry.
W20261118 08:58:36.684237 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 121: this message will repeat every 5th retry.
W20261118 08:58:37.597950 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 121: this message will repeat every 5th retry.
I20261118 08:58:38.418232 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:38.418443 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:39.820638 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 121: this message will repeat every 5th retry.
W20261118 08:58:41.892690 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 121: this message will repeat every 5th retry.
W20261118 08:58:42.113730 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 121: this message will repeat every 5th retry.
W20261118 08:58:43.080662 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 121: this message will repeat every 5th retry.
I20261118 08:58:43.419965 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:43.420174 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:43.757220 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 116: this message will repeat every 5th retry.
W20261118 08:58:45.454900 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 116: this message will repeat every 5th retry.
I20261118 08:58:46.546798 5502 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:46.547006 5502 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:58:46.547993 5502 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:58:46.548974 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:58:46.549110 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:58:46.549193 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:46.549408 5502 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:46.639245 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 126: this message will repeat every 5th retry.
W20261118 08:58:47.498363 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 126: this message will repeat every 5th retry.
I20261118 08:58:48.422314 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:48.422570 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:58:49.046934 5562 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:49.047134 5562 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:49.047889 5562 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:49.048929 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:58:49.049073 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:58:49.049204 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:49.049477 5562 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:49.315145 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 126: this message will repeat every 5th retry.
I20261118 08:58:49.461073 5572 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:58:49.461298 5572 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:49.462234 5572 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:58:49.463202 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:58:49.463716 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:58:49.463868 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:49.464308 5572 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:58:49.506397 5583 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:49.506636 5583 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:49.507725 5583 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:49.508646 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:58:49.510548 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:49.510771 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:58:49.510913 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:49.511158 5583 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:58:51.696516 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 126: this message will repeat every 5th retry.
I20261118 08:58:52.517630 5623 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:58:52.517808 5623 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:52.518447 5623 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:58:52.519479 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:58:52.519641 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:58:52.519788 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:52.520040 5623 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:58:52.539234 5562 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:58:52.539469 5562 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:58:52.540021 5562 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:58:52.540970 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:58:52.541103 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:58:52.541220 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:58:52.541451 5562 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:58:52.667519 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 126: this message will repeat every 5th retry.
W20261118 08:58:53.138473 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 126: this message will repeat every 5th retry.
I20261118 08:58:53.424280 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:53.424507 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:53.622447 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 121: this message will repeat every 5th retry.
W20261118 08:58:55.350610 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 121: this message will repeat every 5th retry.
W20261118 08:58:57.431314 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 131: this message will repeat every 5th retry.
W20261118 08:58:57.928843 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 131: this message will repeat every 5th retry.
I20261118 08:58:58.426076 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:58:58.426276 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:58:58.868916 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 131: this message will repeat every 5th retry.
I20261118 08:58:58.879704 5718 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:58:58.879882 5718 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:58:58.880546 5718 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:58:58.881071 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:58:58.882385 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:58:58.882579 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:58:58.882686 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:58:58.882895 5718 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:59:01.567564 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 131: this message will repeat every 5th retry.
I20261118 08:59:02.426470 5773 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:02.426652 5773 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:02.427489 5773 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:02.428565 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:59:02.428753 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:59:02.428864 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:02.429086 5773 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:02.976280 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 131: this message will repeat every 5th retry.
I20261118 08:59:03.427755 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:03.427986 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:03.493309 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 126: this message will repeat every 5th retry.
W20261118 08:59:03.526296 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 131: this message will repeat every 5th retry.
I20261118 08:59:03.842165 5773 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:03.842361 5773 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:59:03.843048 5773 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:59:03.844028 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:59:03.844183 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:59:03.844280 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:03.844485 5774 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:59:04.585103 5814 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:59:04.585306 5814 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:59:04.586090 5814 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:59:04.587074 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:59:04.588922 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:59:04.589155 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:59:04.589300 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:59:04.589541 5814 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:59:04.966760 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 126: this message will repeat every 5th retry.
W20261118 08:59:07.393180 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 136: this message will repeat every 5th retry.
W20261118 08:59:08.357748 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 136: this message will repeat every 5th retry.
I20261118 08:59:08.429478 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:08.429675 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20261118 08:59:08.546449 5773 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:08.546638 5773 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:08.547279 5773 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:08.548290 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:59:08.548434 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:59:08.548518 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:08.548717 5773 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:59:09.048475 5881 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:59:09.048702 5881 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:59:09.049474 5881 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:59:09.052294 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:59:09.052461 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:59:09.052603 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:59:09.052891 5881 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:59:09.093204 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 136: this message will repeat every 5th retry.
I20261118 08:59:09.637414 5885 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:59:09.637576 5885 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:09.638411 5885 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:59:09.639358 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:59:09.639535 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:59:09.639660 1012 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:09.639889 5885 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:59:10.235421 5903 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:59:10.235605 5903 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:59:10.236227 5903 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:59:10.237162 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20261118 08:59:10.238152 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:59:10.238327 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:59:10.238447 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:59:10.238684 5903 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:59:11.955576 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 136: this message will repeat every 5th retry.
I20261118 08:59:12.268244 5822 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:12.268438 5822 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:12.269086 5822 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:12.270097 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:59:12.270241 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:59:12.270324 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:12.270519 5822 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:12.760205 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 136: this message will repeat every 5th retry.
I20261118 08:59:13.431155 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:13.431352 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:13.441566 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 131: this message will repeat every 5th retry.
I20261118 08:59:13.466651 5822 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:13.466894 5822 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:59:13.467643 5822 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:59:13.468776 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:59:13.468936 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:59:13.469044 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:13.469295 5822 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:13.479260 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 136: this message will repeat every 5th retry.
I20261118 08:59:14.239529 5774 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:14.239713 5774 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:14.240415 5774 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:14.241333 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:59:14.241468 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:59:14.241549 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:14.241740 5774 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:15.662703 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 131: this message will repeat every 5th retry.
W20261118 08:59:16.649622 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 141: this message will repeat every 5th retry.
I20261118 08:59:18.432866 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:18.433055 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:18.799332 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 141: this message will repeat every 5th retry.
W20261118 08:59:18.961671 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 141: this message will repeat every 5th retry.
I20261118 08:59:19.937129 5773 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:19.937314 5773 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:19.938084 5773 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:19.938966 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:59:19.939101 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:59:19.939180 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:19.939373 5773 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:22.930121 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 141: this message will repeat every 5th retry.
W20261118 08:59:23.289916 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 141: this message will repeat every 5th retry.
I20261118 08:59:23.434757 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:23.434917 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:23.452252 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 141: this message will repeat every 5th retry.
W20261118 08:59:23.612715 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 136: this message will repeat every 5th retry.
I20261118 08:59:24.716600 6104 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:24.716773 6104 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:59:24.717419 6104 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:59:24.718353 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:59:24.718483 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:59:24.718606 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:24.718817 6104 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:25.426488 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 136: this message will repeat every 5th retry.
W20261118 08:59:26.145167 1256 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (2053 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:59:26.207154 6145 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:26.207489 6145 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:26.208187 6145 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:26.208881 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:59:26.209022 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:59:26.209111 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:26.209309 6145 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:26.525099 983 heartbeater.cc:648] Failed to heartbeat to 127.0.72.188:41259 (626 consecutive failures): Network error: Failed to ping master at 127.0.72.188:41259: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:59:26.589147 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 146: this message will repeat every 5th retry.
W20261118 08:59:26.840677 1122 heartbeater.cc:648] Failed to heartbeat to 127.0.72.190:42341 (668 consecutive failures): Network error: Failed to ping master at 127.0.72.190:42341: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:59:27.521755 6164 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:59:27.521924 6164 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:27.522910 6164 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:59:27.523799 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:59:27.523941 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:59:27.524058 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:27.524503 6164 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:59:27.950042 6171 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:59:27.950233 6171 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:59:27.951045 6171 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:59:27.951957 2640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:59:27.953095 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
W20261118 08:59:27.953215 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
I20261118 08:59:27.953303 2639 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:59:27.953549 6171 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20261118 08:59:28.436081 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:28.436249 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:28.542209 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 146: this message will repeat every 5th retry.
W20261118 08:59:28.685345 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 146: this message will repeat every 5th retry.
I20261118 08:59:29.914248 6196 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:29.914433 6196 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:29.915042 6196 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:29.916085 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:59:29.916234 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:59:29.916349 1147 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:29.916597 6196 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:32.528707 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 141: this message will repeat every 5th retry.
W20261118 08:59:32.575800 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 146: this message will repeat every 5th retry.
I20261118 08:59:33.228451 6243 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:33.228667 6243 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:33.229346 6243 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:33.230394 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:59:33.230521 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:59:33.230650 1146 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:33.230899 6243 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:33.243453 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 146: this message will repeat every 5th retry.
W20261118 08:59:33.397771 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 146: this message will repeat every 5th retry.
I20261118 08:59:33.437693 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:33.437855 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:34.672181 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 141: this message will repeat every 5th retry.
W20261118 08:59:36.994298 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 151: this message will repeat every 5th retry.
I20261118 08:59:37.215531 6278 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:37.215731 6278 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:59:37.216572 6278 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:59:37.217518 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:59:37.217672 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:59:37.217787 1147 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:37.218015 6278 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20261118 08:59:38.439198 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:38.439344 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:39.181041 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 151: this message will repeat every 5th retry.
W20261118 08:59:39.234125 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 151: this message will repeat every 5th retry.
I20261118 08:59:41.682808 6394 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:59:41.682978 6394 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:59:41.683753 6394 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:59:41.684640 2639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20261118 08:59:41.685710 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:59:41.685855 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:59:41.685954 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:59:41.686161 6394 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:59:42.234537 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 146: this message will repeat every 5th retry.
W20261118 08:59:42.300614 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 151: this message will repeat every 5th retry.
W20261118 08:59:43.085206 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 151: this message will repeat every 5th retry.
I20261118 08:59:43.440968 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:43.441165 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:44.173602 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 151: this message will repeat every 5th retry.
I20261118 08:59:44.314772 6280 raft_consensus.cc:499] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:44.314996 6280 raft_consensus.cc:521] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:44.315663 6280 leader_election.cc:290] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:44.316654 1146 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
W20261118 08:59:44.316818 1147 leader_election.cc:343] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 48a6e2d96fe442b6a84b4773993b4f54
I20261118 08:59:44.316941 1147 leader_election.cc:304] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:44.317176 6280 raft_consensus.cc:2771] T 48a6e2d96fe442b6a84b4773993b4f54 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:44.949927 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 146: this message will repeat every 5th retry.
I20261118 08:59:45.331307 6448 raft_consensus.cc:499] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 78d4026bc230421ea10b8c68a9ef7871)
I20261118 08:59:45.331562 6448 raft_consensus.cc:521] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d4026bc230421ea10b8c68a9ef7871" member_type: VOTER last_known_addr { host: "127.0.72.190" port: 42341 } } peers { permanent_uuid: "857a577b88ba46cc846ecd1a25a722ca" member_type: VOTER last_known_addr { host: "127.0.72.189" port: 33939 } } peers { permanent_uuid: "6c7d9f52503f41e4bafe2821e6fd6af1" member_type: VOTER last_known_addr { host: "127.0.72.188" port: 41259 } }
I20261118 08:59:45.332829 6448 leader_election.cc:290] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341), 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259)
W20261118 08:59:45.341427 2639 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6c7d9f52503f41e4bafe2821e6fd6af1 (127.0.72.188:41259): Network error: Client connection negotiation failed: client connection to 127.0.72.188:41259: connect: Connection refused (error 111)
W20261118 08:59:45.342938 2640 leader_election.cc:336] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 78d4026bc230421ea10b8c68a9ef7871 (127.0.72.190:42341): Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111)
I20261118 08:59:45.343056 2640 leader_election.cc:304] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 857a577b88ba46cc846ecd1a25a722ca; no voters: 6c7d9f52503f41e4bafe2821e6fd6af1, 78d4026bc230421ea10b8c68a9ef7871
I20261118 08:59:45.343267 6448 raft_consensus.cc:2771] T 00000000000000000000000000000000 P 857a577b88ba46cc846ecd1a25a722ca [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20261118 08:59:46.490533 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 156: this message will repeat every 5th retry.
I20261118 08:59:46.903807 6479 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:59:46.903965 6479 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:46.904706 6479 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:59:46.905514 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:59:46.905635 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:59:46.905757 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:46.905967 6479 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:59:48.442613 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:48.442806 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:48.678619 1146 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 156: this message will repeat every 5th retry.
W20261118 08:59:48.863859 1147 consensus_peers.cc:597] T d7f4643e2be049d7b2e3249a40c6b2ac P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f4643e2be049d7b2e3249a40c6b2ac. This is attempt 156: this message will repeat every 5th retry.
I20261118 08:59:49.659436 6508 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:49.659582 6508 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:49.660073 6508 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:49.660934 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:59:49.661060 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:59:49.661172 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:49.661383 6508 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:52.774825 1147 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 156: this message will repeat every 5th retry.
W20261118 08:59:52.820739 1147 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 151: this message will repeat every 5th retry.
W20261118 08:59:53.098852 1147 consensus_peers.cc:597] T 3e292319f0c44e34b49ae6d579df2ecb P 3934fbc351ee4a99add35742e90dfa68 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e292319f0c44e34b49ae6d579df2ecb. This is attempt 156: this message will repeat every 5th retry.
I20261118 08:59:53.444140 2617 master_runner.cc:187] Error getting master registration for 127.0.72.190:42341: Network error: Client connection negotiation failed: client connection to 127.0.72.190:42341: connect: Connection refused (error 111),
I20261118 08:59:53.444365 2617 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20261118 08:59:53.546476 1012 consensus_peers.cc:597] T fc173583a9584741b28f2c0203504a3a P 213b5f7a64e74b109a3a9387e5975042 -> Peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Couldn't send request to peer ad2789ff5b54489c84966cff78be4721. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc173583a9584741b28f2c0203504a3a. This is attempt 156: this message will repeat every 5th retry.
I20261118 08:59:54.863211 6508 raft_consensus.cc:499] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:54.863432 6508 raft_consensus.cc:521] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } }
I20261118 08:59:54.864207 6508 leader_election.cc:290] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649), ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287)
W20261118 08:59:54.865281 1147 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
W20261118 08:59:54.865392 1146 leader_election.cc:343] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: 8d92c02e29724e80820a7114d8bbdf7a
I20261118 08:59:54.865527 1146 leader_election.cc:304] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:54.865759 6508 raft_consensus.cc:2771] T 8d92c02e29724e80820a7114d8bbdf7a P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:55.022512 1146 consensus_peers.cc:597] T 4eecc49bfe014d1d86c84e8a4f16a5bf P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eecc49bfe014d1d86c84e8a4f16a5bf. This is attempt 151: this message will repeat every 5th retry.
I20261118 08:59:55.787971 6508 raft_consensus.cc:499] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad2789ff5b54489c84966cff78be4721)
I20261118 08:59:55.788156 6508 raft_consensus.cc:521] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:55.788782 6508 leader_election.cc:290] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649)
W20261118 08:59:55.789942 1147 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
W20261118 08:59:55.790118 1146 leader_election.cc:343] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Not found: Tablet not found: d92e535f2bf44815b6c003b56dd25b79
I20261118 08:59:55.790246 1146 leader_election.cc:304] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3934fbc351ee4a99add35742e90dfa68; no voters: 213b5f7a64e74b109a3a9387e5975042, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:55.790508 6508 raft_consensus.cc:2771] T d92e535f2bf44815b6c003b56dd25b79 P 3934fbc351ee4a99add35742e90dfa68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20261118 08:59:56.114533 1146 consensus_peers.cc:597] T 39e7203496e0458687be52c3fadd15ea P 3934fbc351ee4a99add35742e90dfa68 -> Peer 213b5f7a64e74b109a3a9387e5975042 (127.0.72.130:33649): Couldn't send request to peer 213b5f7a64e74b109a3a9387e5975042. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7203496e0458687be52c3fadd15ea. This is attempt 161: this message will repeat every 5th retry.
I20261118 08:59:56.823601 6599 raft_consensus.cc:499] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261118 08:59:56.823793 6599 raft_consensus.cc:521] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad2789ff5b54489c84966cff78be4721" member_type: VOTER last_known_addr { host: "127.0.72.129" port: 39287 } } peers { permanent_uuid: "213b5f7a64e74b109a3a9387e5975042" member_type: VOTER last_known_addr { host: "127.0.72.130" port: 33649 } } peers { permanent_uuid: "3934fbc351ee4a99add35742e90dfa68" member_type: VOTER last_known_addr { host: "127.0.72.131" port: 46447 } }
I20261118 08:59:56.824525 6599 leader_election.cc:290] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287), 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447)
W20261118 08:59:56.825538 1012 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ad2789ff5b54489c84966cff78be4721 (127.0.72.129:39287): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
W20261118 08:59:56.825711 1011 leader_election.cc:343] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3934fbc351ee4a99add35742e90dfa68 (127.0.72.131:46447): Not found: Tablet not found: 380fa8ff3e2f428b81b7567bc7bcefee
I20261118 08:59:56.825819 1011 leader_election.cc:304] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 213b5f7a64e74b109a3a9387e5975042; no voters: 3934fbc351ee4a99add35742e90dfa68, ad2789ff5b54489c84966cff78be4721
I20261118 08:59:56.826076 6599 raft_consensus.cc:2771] T 380fa8ff3e2f428b81b7567bc7bcefee P 213b5f7a64e74b109a3a9387e5975042 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261118 08:59:56.949539 2644 maintenance_manager.cc:419] P 857a577b88ba46cc846ecd1a25a722ca: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1964 bytes on disk
I20261118 08:59:56.954752 2643 maintenance_manager.cc:643] P 857a577b88ba46cc846ecd1a25a722ca: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":340,"thread_start_us":525,"threads_started":2,"wal-append.queue_time_us":421}
W20261118 08:59:57.664301 1264 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
W20261118 08:59:57.664523 1264 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
W20261118 08:59:57.664608 1264 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
W20261118 08:59:57.664665 1264 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
F20261118 08:59:57.675375 1297 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
*** Check failure stack trace: ***
*** Aborted at 1794992397 (unix time) try "date -d @1794992397" if you are using GNU date ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681154 1288 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681181 1291 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
*** SIGABRT (@0x3e800000122) received by PID 290 (TID 0x7f0cbcf49700) from PID 290; stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681154 1288 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681181 1291 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.682323 1293 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7f0cdc494980 (unknown) at ??:0
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681154 1288 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681181 1291 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.682323 1293 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683583 1281 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681154 1288 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681181 1291 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.682323 1293 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683583 1281 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683658 1279 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_52bba9536ae942a59235d8087d61576b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7f0cd5204e87 gsignal at ??:0
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681154 1288 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681181 1291 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.682323 1293 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683583 1281 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683658 1279 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_52bba9536ae942a59235d8087d61576b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683946 1282 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681154 1288 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681181 1291 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.682323 1293 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683583 1281 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683658 1279 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_52bba9536ae942a59235d8087d61576b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683946 1282 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683988 1285 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
F20261118 08:59:57.675616 1289 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675695 1284 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675706 1283 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675868 1277 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_8f6feb81c1374d42bc1e0acb856b75cd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.675885 1272 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9dbf0bcdd2ac45978f8b2c56be77c11a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677057 1296 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677203 1273 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_738b3d8aface40768cac1e7b01d47046 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677455 1280 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_3655f04cfdf84b26839c6e76aac6ebbe on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677484 1278 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_bfd124a61a904e60bc99bee8716f64cc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677584 1287 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.677711 1286 master-stress-test.cc:353] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.678719 1275 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_82d1541f550a4d5482cff56810c97a22 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679875 1271 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_0327d29a2f19450b872c54c554d7654e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1292 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.679920 1276 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_903cefd8f2a74c89be4bf41b84d26ba4 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681154 1288 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.681181 1291 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.682323 1293 master-stress-test.cc:393] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683583 1281 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683658 1279 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_52bba9536ae942a59235d8087d61576b on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683946 1282 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.683988 1285 master-stress-test.cc:330] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)F20261118 08:59:57.684332 1274 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b5863ec6b7e144f58594f23a0634bf0a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 771) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.72.190:42341,127.0.72.189:33939,127.0.72.188:41259, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7f0cd52067f1 abort at ??:0
@ 0x7f0cd7f53d5d google::LogMessage::Fail() at ??:0
@ 0x7f0cd7f57b23 google::LogMessage::SendToLog() at ??:0
@ 0x7f0cd7f5375c google::LogMessage::Flush() at ??:0
@ 0x7f0cd7f54ee9 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40b4d5 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:393
@ 0x447484 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44733d std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x4472fa _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x4472ae std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x446fe9 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f0cd588e6df (unknown) at ??:0
@ 0x7f0cdc4896db start_thread at ??:0
@ 0x7f0cd52e761f clone at ??:0